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PREFACE 


This  report  was  created  for  the  F-16  Aircrew  Training  De¬ 
velopment  Project  contract  no.  P02604-79-C8875  for  the  Tactical 
Air  Command  to  comply  with  the  requirements  of  CDRL  nos.  B036r 
d038f  and  B047.  The  project  entailed  the  design  and  development 
of  an  instructional  system  for  the  F-16  RTU  and  instructor 
pilots.  During  the  course  of  the  project,  a  series  of  develop¬ 
ment  reports  was  issued  describing  processes  and  products.  A 
list  of  those  reports  follows  this  page.  The  user  is  referred  to 
Report  No.  34,  A  Users  Guide  to  the  F-16  Training  Development 
Reports,  for  an  overview  and  explanation  of  the  series,  and 
Report  No.  35,  F-16  Final  Report,  for  an  overview  of  the 
Instructional  System  Development  Project. 
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EXECUTIVE  SUMMARY 


t 

b 

Media  selection  is  the  process  of  minimizing  the  costs  due 
to  media  acquisition,  use,  or  maintenance  while  maximizing  the 
likelihood  of  consistent  instructional  effectiveness.  Media 
selection  for  aircrew  training  involves  two  major  categories  of 
med  ia : 

1.  Media  intended  for  use  in  learning  centers,  ranging 
from  workbooks  to  computer-assisted  instruction.  .  -f 

2.  Simulator  or  training  devices  which  are  intended  to 

V  provide  •hands-on*  practice. 


Factors  which  must  be  considered  in  the  media  selection 
process  include  cost,  ability  of  a  device  to  implement  specific 
instructional  strategies,  motivating  capabilities,  and  the  pop¬ 
ularity  of  the  media  device  with  the  student  population.  Some  of 
the  constraints  to  be  considered  in  the  F-16  environment  include 
the  necessity  of  premature  media  decisions,  the  late  arrival  of 
training  devices,  the  varying  availability  of  media  cross  train¬ 
ing  sites,  the  exisiting  media  production  facilities,  the  tenta¬ 
tive  nature  of  the  instructional  content,  the  required  mixture  of 
type  *A*  and  aircraft,  and  the  varying  availability  of  flight 
time  across  training  sites.  — ' 

The  media  selection  process  begins  with  a  complete  list  of 
the  criterion-referenced  objectives  (CROs)  from  the  objectives 
hierarchy  analysis.  These  objectives  are  divided  into  academic 
(i.e.,  learning  center  learning  activities)  and  training  device 
categories,  and  then  classified  in  terms  of  taxonomic  levels. 
Instructional  strategies  appropriate  for  each  level  of  learning 
required  are  then  specified. 

Associated  with  each  strategy  is  a  media  requirements 
profile.  All  constraints  affecting  media  selection  for  each  ob¬ 
jective  are  also  specified.  The  available  media  are  listed,  and 
those  media  which  meet  the  strategy  requirements  and  constraints 
are  retained.  The  instructional  requirements  for  each  objective 
are  then  identified  and  alternative  media  configurations  are 
determined.  The  production,  use,  and  maintenance  costs  associat¬ 
ed  with  each  alternative  are  estimated.  A  media  decision  model 
is  then  used  to  identify  alternative  media  for  each  objective. 

The  total  costs  of  each  media  assignment  alternative  are  then 
computed  and  reviewed. 

For  those  objectives  identified  as  requiring  training 
devices,  additional  media  selection  steps  are  required.  It  is 
necessary  to  identify  behavior  elements  contributing  to  task 
difficulty.  These  elements  are  mapped  into  corresponding  levels 
of  training  devices  ranging  from  low  fidelity,  non- interactive 
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devices  (e.g.,  cardboard  mock-ups'  to  high  fidelity,  highly  in¬ 
teractive  devices  (e.g.,  operational  flight  trainers).  The  F-16 
aircraft  is  assigned  only  when  complete  fidelity  is  instruction- 
ally  necessary.  The  possible  configurations  for  each  training 
device  are  specified,  along  with  the  time  requirements  associated 
with  each  configuration.  The  cost  of  each  configuration  is  then 
estimated  and  the  best  configuration  selected. 


■^The  media  selection  process  is  a  dyncimic  one  and  the  pro¬ 
cedures  described  extend  over  the  lifetime  of  the  F-16  training 
system.  To  aid  in  the  standardization  and  docunentation  of  the 
media  selection  process,  a  set  of  forms  are  provided  which  assist 
in  the  data  collection  and  decision-making  steps. 


CONTENTS 


Page 

Preface  .  i 

F-16  Aircrew  Training  Development  Project  Reports  .  ii 

EXECUTIVE  SUMMARY  .  V 

1.0  INTRODUCTION  .  1 

1.1  Purpose .  2 

1.2  Organization  .  3 

2.0  RATIONALE .  4 

2.1  The  Scope  of  Media  Selection  Activities  .  4 

2.2  Media  Selection  in  Relation  to  Other 

Instructional  Development  Processes  .  5 

2.3  Factors  in  Media  Selection  .  6 

2.4  Constraints  Upon  Media  Selection  .  10 

3.0  MEDIA  SELECTION  PROCESS  .  14 

3.1  Step  1:  Sort  Objectives  into  Academic 

and  Training  Device  Groups  .  19 

3.2  Step  2:  Classify  Academic  Objectives  .  19 

3.3  Step  3;  Specify  Instructional  Strategies 

and  Alternates . 22 

3.4  Step  4:  State  All  Nonstrategy  Factors 

Affecting  Media  Selection  .  22 

3.5  Step  5:  List  the  Initial  Media  Pool . 24 


3.6  Step  6:  Eliminate  and  Reconfigure  Media  to 

Produce  the  Acceptable  Media  List  ....  24 

3.7  Step  7;  Determine  Special  Instructional 

Media  Requirements  for  Each  Objective  .  .  27 


3.8  Step  8:  Determine  Alternate  Media 

Assignment  Patterns  .  28 

3.9  Step  9:  Determine  Media  Production  and 

Maintenance  Costs  .  30 


vii 


CONTENTS  (cont.) 


3.10 

3.11 

3.12 

3.13 

3.14 

3.15 

3.16 

3.17 

3.18 

3.19 

3.20 

3.21 

3.22 

3.23 

3.24 

3.25 


Page 


Step 

10: 

Construct  the  Media  Decision 

Models  . 

Step 

11: 

Select  Media  for  All  Objectives 

Under  Each  Assignment  Pattern  .... 

Step 

12: 

Determine  the  Total  Cost  of  Each 

Media  Assignment  Pattern  . 

Step 

13: 

Review  Cost  Tradeoffs  Related  to 

Each  Assignment  Pattern  . 

Step 

14: 

Summarize  Media  Resource 

Requirements  . 

Step 

15: 

Analyze  Behavior  Elements 

Contributing  to  Task  Difficulty  ,  .  . 

Step 

16: 

Specify  Isolated  Training 

Difficulty  Thresholds . 

Step 

17: 

Conduct  Extended  Task  Analysis  .  .  . 

Step 

18: 

Insert  New  Tasks  into  Task  Listing 

Step 

19: 

Specify  Training  Devices  . 

Step 

20; 

Assign  Tasks  to  Training  Devices  .  . 

Step 

21; 

Specify  Possible  Configurations  for 
Each  Training  Device  Type  . 

Step 

22; 

Determine  Instructional  Strategies 
to  be  Used  in  Each  Training  Device 
and  the  Potential  Instructional 
Features  for  Each  Training  Device  .  . 

Step 

23; 

Determine  Performance  Measurement 
to  be  Executed  in  Training  Devices 
and  the  Potential  Performance 
Measurement  Features  for  Each 

Training  Device  . 

Step 

24: 

Estimate  the  Cost  of  Each  Configura¬ 
tion  for  Each  Training  Device  .... 

Step 

25; 

Determining  Time  Requirements 
for  Each  Configuration  of  Each 

Training  Device  . 

viii 


30 

30 

33 

33 

33 

34 

36 

36 

36 

37 

38 

43 


43 


46 

46 


48 


CONTENTS  (cont.) 


Page 


3.26 

Step 

26: 

Consolidate  Configurations  Using 
Time  Factors  . 

•  • 

49 

3.27 

Step 

27: 

Determine  Cost  for  Total  Training 
Device  Plan  . 

•  • 

50 

3.  28 

Step 

28: 

Adjust  Assignments  for  Best  Total 
Training  Device  Plan  Cost  .... 

•  • 

51 

3.29 

Step 

29: 

Write  Training  Device  Specifi¬ 
cations  . 

•  • 

51 

3.30 

Step 

30: 

Collect  Time  to  Proficiency  and 
Level  of  Use  Data  on  Training 
Devices . . 

•  • 

51 

3.31 

Step 

31: 

Adjust  Training  Device  Selection 
Data  Base . 

•  • 

52 

References .  53 

Attachment  I;  P-16  ACADEMIC  OBJECTIVE  CATEGORIES 

AND  INSTRUCTIONAL  STRATEGIES  .  I-l 


lx 


LIST  OF  FIGURES  AND  TABLES 


Page 


FIGURES 

1.  Media-Related  Costs  .  9 

2.  Flow  Diagram  of  the  Media  Selection  Model .  17 

3.  Device  Type  Classifications  .  39 

TABLES 

1.  Academic  Objective  Analysis  Worksheet  .  20 

2.  Strategy  Media  Requirements  Profile  .  23 

3.  Media  Elimination  Table  .  26 

4.  Media  Assignment  Pattern  Record  Form  .  29 

5.  Media  Selection  Record  Form  .  31 

6.  A  Media  Summary  Table .  32 

7.  Task  Element  Difficulty  Rating  Sheet .  35 

8.  Task  Listing  Data  Worksheet  . .  41 

9.  Training  Device  Configuration  Record  .  44 

10.  Training  Device  Configuration  Worksheet  .  45 

11.  Table  for  Recording  Instructional  Strategies 

and  Features  for  Each  Training  Device .  47 


X 


F-16  MEDIA  SELECTION 
AND  UTILIZATION  PLAN  REPORT 


1 . 0  INTRODUCTION 


Media  selection  is  a  process  of  minimizing  costs  attendant 
to  media  acquisition,  use,  and  maintenance  while  maximizing 
instructional  effectiveness.  It  is  an  engineering  process  in 
which  tiie  benefits  which  can  be  realized  through  application  of 
learning  principles  and  instructional  research  are  balanced 
against  the  lessons  of  experience  and  economics  in  designing 
real-world  instructional  systems  and  providing  for  the  necessary 
equipment  and  instructional  materials. 

Media  selection  is  one  stage  of  the  instructional  develop¬ 
ment  process  which  is  partly  administrative  in  nature  and  partly 
governed  by  the  principles  of  psychology  and  learning.  On  one 
hand  it  uses  factor  optimization  decision-making  techn.ques  to 
produce  a  "best"  media  decision  in  the  presence  of  conflicting 
goals.  This  resolution  of  the  conflicting  demands  for  instruc¬ 
tional  sophistication,  economy,  and  ease  of  administration  is 
very  much  an  administrative  concern.  On  the  other  hand  media 
selection  must  take  into  account  the  creative  application  of  the 
principles  of  instructional  strategy  and  materials  development 
that  allow  it  to  happen.  Without  a  judgment  along  this  dimen¬ 
sion,  two  competing  media  selection  plans  must  be  assumed  equally 
effective,  regardless  of  the  sophistication  of  the  media  devices 
chosen . 

Few  processes  affect  an  instructional  system's  cost  and 
practical  utility  as  profoundly  as  media  selection.  The  develop¬ 
ment  of  an  instructional  system  for  aircrew  training  commonly 
involves  the  expenditure  of  thousands  of  dollars  for  learning 
center  media,  the  expenditure  of  still  other  thousands  of  dollars 
over  the  lifetime  of  the  system  on  the  development  and  update  of 
mediated  instruction,  and  multi-millions  of  dollars  for  aircraft 
simulation  training  devices.  Decisions  made  during  the  media 
selection  process  have  a  tremendous  financial  impact. 

In  direct  proportion  to  the  economic  importance  of  media 
selection  as  a  process  is  the  frustration  and  confusion  caused 
for  instructional  developers  and  their  clients  because  of  its 
many  unresolved  issues.  These  frustrations  arise  from  the  fact 
that  many  factors  are  balanced  against  each  other  in  the  making 
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of  media  selection  decisions  and  there  are  few,  if  any,  firm 
guidelines  for  determining  media  which  are  absolutely  required 
for  a  given  instructional  goal.  A  large  volume  of  research  has 
been  conducted  to  determine  the  superiority  of  certain  media  over 
others,  with  no  conclusive  results.  In  fact,  media  specialists 
themselves  do  not  agree  on  which  media  are  preferred  for  which 
application,  and  there  seem  to  be  few  selections  which  cannot  be 
rationalized  away  by  some  form  of  argument.  On  the  contrary, 
instructional  developers  with  creative  imaginations  can  often 
invent  ways  to  reconfigure  media  devices  and  training  sequences 
to  achieve  approximately  equal  effectiveness  from  different 
media.  The  problems  of  media  selection  have  had  so  few  definite 
answers  that  many  prominent  texts  on  instructional  development 
avoid  the  topic  altogether.  Yet,  in  view  of  the  obvious  effects 
of  media  selection  on  training  budgets,  the  problems  and  uncer¬ 
tainties  of  media  selection  cannot  be  ignored. 


1.1  Purpose 

The  purpose  of  this  paper  is  to  outline  a  method  and  ration¬ 
ale  for  the  selection  of  both  learning  center  and  training  device 
media  for  use  in  the  F-16  instructional  system.  A  report 
containing  actual  media  selection  decisions  will  be  issued  as 
project  reports  no.  24,  "F-16  Media  Selections  and  Syllabi"  and 
no.  26,  "F-16  Instructor/Course  Managers  Media  Selection  and 
Syllabus  Report." 

General  decisions  and  policies  made  during  the  F-16  media 
selection  process  and  decisions  relative  to  individual  objectives 
will  be  contained  in  reports  no.  24  and  26,  which  will  be  issued 
upon  completion  of  the  media  selection  and  sequencing  process. 

The  F-16  instructional  developers  have  attempted  to  address 
several  main  problems  which  presently  exist  in  the  media  selec¬ 
tion  process;  (1)  the  problem  of  striking  a  balance  between  the 
instructional  and  economic  factors  which  influence  media  deci¬ 
sions  in  such  a  way  that  one  kind  does  not  suffer  unduly  at  the 
hands  of  the  other,  (2)  the  problem  of  systematizing  considera¬ 
tion  of  these  major  factors  in  the  context  of  an  organized 
process,  and  (3)  the  problems  attendant  to  basing  the  specifica¬ 
tion  of  simulator- type  training  devices  on  bodies  of  data  gener¬ 
ated  during  instructional  development  rather  than  on  more  subjec¬ 
tive  means. 

The  result  is  a  guideline  and  general  procedure  for  the 
media  selection  process  which  may  be  used  by  instructional  system 
personnel  who  possess  a  background  in  instructional  development. 
Though  the  methodology  reported  here  was  developed  in  the  context 
of  instructional  system  development  for  a  single-seat  aircraft, 
it  is  felt  that  minor  adaptations  can  render  the  methodology 
usable  for  multi-person  crews  of  any  number  and  non-aircrew 
training  applications  as  well. 
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1.2  Organization 


The  main  body  of  this  report  is  contained  in  Sections  2.0 
and  3.0.  Section  2.0  is  introductory.  It  outlines  the  media 
selection  problem  and  relates  the  media  selection  process  to 
other  instructional  development  processes.  In  particular,  the 
media  selection  document  is  characterized  as  a  "living”  document 
to  be  maintained  on  a  continual  basis  along  with  several  other 
instructional  system  data  base  documents.  This  maintenance  proce¬ 
dure  insures  that  the  instructional  development  principles  built 
into  the  system  during  its  design  remain  there  during  its  opera¬ 
tion.  This  section  also  names  the  set  of  factors  and  constraints 
which  need  to  be  balanced  during  media  selection.  Section  3.0 
describes  the  media  selection  process.  Modifications  to  this 
basic  procedure  and  the  data  generated  by  the  selection  process 
are  documented  in  Development  Report  No.  26,  "F-16  Pilot  Media 
Selection".  The  final  media  profile  utilized  in  the  F  1600B 
course  is  reported  in  Development  Report  No.  31,  "F-16  Training 
Media  Mix". 
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2.0  RATIONALE 


This  section  relates  the  media  selection  process  to  other 
development,  practical,  and  economic  concerns.  It  begins  with  a 
statement  of  the  scope  of  media  selection  activities,  describes 
how  media  selection  influences  and  is  influenced  by  other 
instructional  development  activities,  and  discusses  the  balancing 
factors  and  constraints  which  operate  within  media  selection. 


2.1  The  Scope  of  Media  Selection  Activities 

Media  selection  for  aircrew  training  consists  of  two  main 
categories  of  activity.  The  first  deals  with  selection  of  media 
for  use  in  learning  centers  for  information  and  knowledge  convey¬ 
ance  and  practice.  This  includes  well-known  standard  media  types 
varying  from  the  very  simple  and  inexpensive  (e.g.,  charts,  work¬ 
sheets,  and  summary  pages)  to  the  complex  and  costly  (e.g., 
computer-assisted  instruction,  cue-see  type  systems,  and  video¬ 
tape  recording) .  It  includes  media  which  ate  intended  for  use  by 
isolated  students  (e.g.,  workbooks,  worksheets,  and  tutorials), 
students  in  small  groups  (e.g.,  discussion  groups  and  problem¬ 
solving  exercises)  and  students  in  large  groups  (e.g.,  lecture). 

A  second  area  of  media  selection  centers  in  the  configura¬ 
tion  of  simulation  training  devices  which  ate  intended  to  create 
to  some  degree  a  real-world-like  performance  environment  in  which 
the  student  may  practice  either  fragmented  or  complete  behaviors. 
Examples  in  this  area  also  range  from  the  simple  (e.g.,  isolated 
realia,  dead  systems,  mock-up  panels,  displays),  to  the  complex 
(e.g.,  operational  flight  trainers,  computer  avionics  simula¬ 
tions,  weapon  system  trainers),  depending  on  the  scope  of  the 
enviromental  factors  being  simulated. 

There  is  much  uncertainty  and  debate  in  the  training  device 
design  area  of  media  selection  because  of  the  growing  complexity 
of  training  devices  and  simulated  environments  and  the  uncer¬ 
tainty  surrounding  the  actual  instructional  effectiveness  of 
given  optional  configurations.  In  the  past  no  systematic  body  of 
research  data  or  detailed  record  has  been  compiled  to  guide 
developers  in  specifiying  training  device  characteristics,  and 
the  practice  of  determining  device  characteristics  in  a  non- 
subjective,  organized  fashion  from  a  statement  of  the  training 
requirements  has  not  been  widely  adopted. 

Because  of  the  indefiniteness  involved  in  both  areas  of 
media  selection  and  the  inability  to  supply  firm  answers  to 
either  learning  center  or  training  device  media  selection  ques¬ 
tions,  the  process  of  selection  is  not  generally  looked  upon  as  a 
very  precise  activity.  Subjective  approaches  to  selection 
abound,  although  some  highly  systematic  methods  have  been  created 
(Braby,  et,  al . ,  1975).  For  the  most  part,  the  problem  faced  by 
the  developer  is  a  matter  of  distinguishing  between  necessities 
and  non-necessities. 
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2.2  Media  Selection  in  Relation  to  Other  Instructional 

Development  Processes 

While  earlier  stages  in  the  instructional  development 
process  are  analytic  phases  which  gather  and  correlate  data  for 
use  by  the  developer,  media  selection  is  a  synthetic,  or  design, 
stage  in  which  plans  are  formulated  which  give  structure  to  the 
instructional  system.  Media  selection  builds  on  the  products  of 
previous  development  stages,  and  takes  place  interactively  with 
other  development  tasks,  therefore  it  is  appropriate  to  describe 
its  relation  to  other  stages  of  instructional  development. 

Early  stages  of  instructional  development  supply  the 
substance  which  the  media  selection  process  works  with.  Media 
selection  for  F-16  occurs  following:  (1)  the  generation  of  a  jcb 
task  inventory  or  task  listing,  (2)  selection  of  a  set  of 
terminal  training  tasks,  (3)  generation  of  a  list  of  difficult- 
to-operationalize  behaviors  called  "goals",  (4)  generation  of 
criterion-referenced  objectives  (CROs)  to  be  used  as  terminal 
performance  measurement  points,  and  (5)  generation  of  objectives 
hierarchies.  The  results  of  these  processes  taken  together  form 
a  data  base  of  instructional  requirements  for  F-16  pilots  (and 
also  for  F-16  instructor  pilots,  since  the  same  data  base  was 
created  for  that  position  as  well) .  The  data  base  documents  are 
used  as  inputs  to  the  media  selection  process.  They  list  the 
instructional  events  for  which  media  needed  to  be  selected. 

Those  foundation  documents  and  the  details  of  their  generation 
are  described  in  separate  F-16  project  reports. 

Concurrent  with  media  selection  and  interacting  with  it,  a 
process  of  syllabus-building  or  "sequencing"  takes  place.  For 
sequencing  the  same  base  of  documentation  from  previous  stages  is 
used  as  an  input.  Sequencing  is  the  process  of  arranging  those 
instructional  events  taken  from  the  CROs  and  objectives  hier¬ 
archies  in  the  order  students  encounter  them  in  the  course  of 
instruction.  Ideally,  the  sequence  (syllabus)  promotes  the  best 
obtainable  gradual  progression  of  student  behavior  from  the  time 
of  entry  into  training  to  the  point  of  mastery  of  all  terminal 
course  behaviors.  Media  selection  and  sequencing  are  closely 
related  to  each  other.  The  order  of  instructional  events  in 
training  often  makes  particular  media  selections  more  practical. 
Media  selection  decisions  determine  the  types  of  instructional 
events  and  environments  which  can  be  used  in  the  sequence. 

Special  attention  in  both  processes  is  given  to  adjusting  the 
student's  instructional  experience  to  within  certain  parameters 
which  the  student  will  find  tolerable  or  agreeable  in  terms  of 
variety,  pace,  redundancy,  and  practical  usability. 

The  results  of  the  selection  and  sequencing  processes  are  a 
course  syllabus  and  a  list  of  individual  media  assignments  by 
instructional  event.  These  documents  lead  to  further  instruc¬ 
tional  system  development  activities,  although  they  represent  by 
themselves  a  significant  addition  to  the  fundamental  data  base 
documentation  referred  to  earlier.  Throughout  the  lifetime  of 


the  instructional  system  these  documents  should  have  a  life  of 
their  own,  changing  as  the  documentation  which  acts  as  input  to 
them  changes  (i.e.,  the  task  listing,  goal  analysis,  CROs,  objec¬ 
tives  hierarchies)  and  in  turn  influencing  other  documents.  An 
instructional  system  which  does  not  maintain  these  documents, 
though  derived  through  the  application  of  good  instructional 
development  practices,  cannot  remain  functional  and  efficient  for 
long . 


The  media  selection  document,  taken  in  conjunction  with  the 
sequencing  document,  serves  several  purposes.  It  serves  as  an 
inventorying  base  during  the  actual  production  of  instructional 
materials.  Prior  to  that  it  serves  as  a  base  for  calculating 
projections  of  the  production  organization  and  costs  which  will 
be  required  during  production  planning.  For  instance,  the  types 
and  numbers  of  instructional  presentations  (called  "segments") 
can  be  determined  only  after  media  selection,  as  well  as  projec¬ 
tion  of  the  number  and  types  of  media  devices  required  to  present 
instruction  for  a  given  volume  of  students.  Indirectly,  these 
figures  lead  to  the  partial  projection  of  numbers  and  types  of 
instructors,  administrators,  and  technicians  who  will  be  requited 
to  staff  the  student  learning  center,  simulator  locations,  media 
equipment  repair  facilities,  and  media  revision,  production,  and 
reproduction  facilities.  All  of  these  items  have  both  short-  and 
long-term  financial  implications.  In  fact,  the  financial  impact 
of  these  decisions  is  the  major  cost  factor  in  the  instructional 
system  operation.  For  a  full  description  of  the  impact  of  sylla¬ 
bus  and  media  selection  costs  on  total  system  cost,  see  project 
report  no.  21,  "F-16  Instructional  System  Cost  Study  Report." 


2.3  Factors  in  Media  Selection 

In  minimizing  the  costs  and  maximizing  the  benefits  of  media 
selections  many  factors  must  be  considered.  Media  selections 
must  be  a  "best"  solution  based  upon  stated  criteria,  the  optimi¬ 
zation  on  a  set  of  sometimes  conflicting  demands.  "Make  it  inex¬ 
pensive"  often  leads  to  different  solutions  than  "make  it  effec¬ 
tive,"  and  frequently  both  disagree  with  "make  it  interesting." 
Ideally,  the  media  selection  process  insures  that  each  selection 
factor  is  properly  considered  in  the  decision-making  process. 

The  media  selection  process  described  in  Section  3.0  of  this 
report  is  structured  in  such  a  way  that  consideration  of  these 
demands  takes  place  in  an  orderly  fashion  and  seeks  to  insure 
that  each  factor  is  considered  in  proportion  to  its  relative 
importance  to  the  development  task  at  hand.  Failure  to  consider 
all  relevant  factors  in  media  selection,  or  undue  and  dispropor¬ 
tionate  emphasis  on  one  factor  can  lead  to  wasteful  spending  or 
can  produce  media  systems  which  are  unpalatable  to  the  student, 
difficult  to  use,  instructionally  ineffective,  or  any  combination 
of  these.  Plentiful  examples  of  each  can  be  found  in  the  history 
of  instructional  development. 


The  major  factors  which  have  been  considered  during  the  F-16 
media  selection  process  are  listed  below  along  with  a  statement 
of  the  relative  priority  of  each  within  the  F-16  context. 


2.3.1  Ability  to  lapleaent  Effective  Instructional 
Strategies 

The  ability  of  a  device  to  implement  a  specific  instruc¬ 
tional  strategy  is  a  factor  of  media  selection  too  often  ignored 
or  diminished  in  importance,  even  though  it  should  logically 
rival  or  even  outweigh  cost  in  determining  which  devices  are 
selected.  One  reason  for  this  is  probably  the  primitive  state  of 
affairs  with  respect  to  instructional  strategy  prescription. 
Psychologists  interested  in  instruction  have  only  recently  turned 
their  attention  from  the  study  wf  how  people  learn  to  the  study 
of  how  to  instruct  and  establish  new  behaviors.  Therefore,  the 
practice  of  specifying  instructional  strategies  is  in  its 
infancy,  and  the  idea  that  media  should  be  constructed  specifi¬ 
cally  to  implement  strategies  is  relatively  new. 

F-16  media  candidates  will  be  selected  following  the 
specification  of  a  set  of  generic  instructional  stragegies  (see 
Attachment  I).  Media  are  preferred  if  they  demonstrate  a 
capability  for  implementing  these  strategies,  and  possible  media 
combinations  and  reconfigurations  are  considered  where  strategies 
can  not  be  accommodated  in  an  attempt  to  produce  media  as  capable 
as  possible  in  implementing  them.  Characteristics  of  a  media 
device  which  take  part  in  the  implementation  of  strategies  are: 

(1)  the  ability  to  generate  appropriate  stimulus  conditions,  (2) 
the  ability  to  receive  student  responses,  (3)  the  ability  to 
process  those  responses,  and  (4)  the  ability  to  provide 
appropriate  counter-responses  (feedback  and/or  subsequent 
stimulus  conditions).  Depending  upon  the  strategy  being  used, 
these  requirements  may  be  simple  or  complex.  They  may  call  for 
the  ability  to  present  simple  pictorial,  detailed  photographic, 
or  complex  moving  computer-generated  graphics  as  stimuli.  They 
may  call  for  the  reception  of  responses  from  the  student  which 
are  simple  yes/  no  answers,  or  complex  verbal  expressions,  or 
coordinated  manipulation  of  aircraft-like  controls.  They  may 
call  for  response  processing  and  feedback  which  ranges  from 
simple  correct  answer  comparisions  followed  by  knowledge  of 
results,  to  complex  computerized  calculation  routines  followed  by 
simulated  aircraft  response.  In  addition,  all  of  these 
characteristics  may  be  required  under  timed  conditions  and  at 
varying  levels  of  fidelity  to  real-world  performance 
env i ronments  . 


2.3.2  Cost 

The  cost  factor  is  of  extreme  importance  in  media  selections 
because  cost  is  most  often  the  only  upward  limiting  factor.  This 
is  true  of  the  F-16  project.  Media  selections  entail  not  only 
immediate  costs  but  delayed  costs,  some  of  which  continue 
throughout  the  lifetime  of  the  media  system. 
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Figure  1  summarizes  the  costs  associated  with  media  device 
procurement  and  operation.  The  table  identifies  media  costs 
during  five  main  time  frames:  (1)  during  the  design  of  media 
systems  and  instructional  materials,  (2)  during  the  procurement 
of  a  media  system,  (3)  during  development  of  the  media  system  and 
instructional  materials,  (4)  during  initial  implementation  of  the 
system,  and  (5)  during  ongoing  use  of  the  media  system  and 
materials.  It  identifies  major  categories  also,  under  which 
expenses  might  occur  during  each  phase. 

Media  vary  widely  in  the  exact  values  of  each  entry  in  this 
table.  Even  for  the  same  medium,  expenses  vary  with  differing 
training  conditions,  such  as  already-avai lable  resources,  differ¬ 
ent  geographic  locations,  differences  in  manpower  availability, 
and  differing  organizational  patterns.  Not  all  media  require  all 
of  the  listed  expenses,  but  all  media  will  have  a  subset  of  the 
expenses.  The  more  sophisticated  the  medium,  the  more  expenses 
are  likely  to  be  attached,  in  both  the  short  and  the  long  terms. 
For  learning  center  media  the  acquisition  costs  are  comparatively 
minor,  but  the  maintenance  costs  for  instructional  materials  can 
be  substantial  for  aircrew  training.  For  training  devices  acqui¬ 
sition  costs  are  often  very  large,  even  more  so  when  multiple 
training  sites  and  therefore  multiple  copies  of  a  device  are 
called  for. 


2.3.3  Attention-  and  Interest-generating  Qualities 

Since  media  devices  and  the  presentations  made  by  them 
represent  a  main  communication  link  during  instruction,  it  is 
important  not  only  that  the  message  be  clearly  stated  but  that 
the  presentation  arouse  attention  and  interest  in  the  user. 
Inherent  in  most  instructional  strategies  is  an  interaction 
between  student  and  device  that  require  active  participation  by 
the  student.  This  requirement  for  interaction  can  serve  as  an 
attention-generating  tool.  In  addition,  if  the  mode  of  interac¬ 
tion  is  interesting  to  the  student  or  facilitates  a  student's 
personal  learning  goal,  it  can  act  as  an  interest-generating  tool 
as  well . 

Many  media  selection  decisions  are  made  without  considering 
the  impact  of  the  decision  upon  student  attention  and  interest. 
Still  others  are  mode  upon  incorrect  assumptions  of  what  the 
developer  thinks  will  be  attention-getting  or  interesting  to  the 
student  rather  than  upon  data  from  student  trial  and  reactions. 
Results  of  such  selections  are  media  selections  whicli:  (1) 

offend  students  because  they  are  inappropriate  to  student's  taste 
and  learning  styles,  (2)  bore  students  because  they  operate  too 
slowly,  because  the  student  has  no  opportunity  to  interact,  or 
because  there  is  too  much  use  made  of  one  medium,  (3)  are  a 
distraction  to  students  because  they  require  too  much  attention 
just  for  operation,  or  (4)  are  too  fast  for  students'  learning 
pace  and  cannot  be  controlled  by  the  student,  thus  forcing  the 
student  to  move  at  the  presentation's  rate  or  repeat  the  instruc- 
tion--which  some  media  do  not  allow. 


8 


2.3.4  Use  of  Standard  Systems 

Several  potential  benefits  encourage  utilization  of  exist¬ 
ing,  standard  media  systems  where  possible.  Procurement  of 
devices,  maintenance  service,  acquisition  of  spare  parts,  and 
acquisition  of  replacement  devices  is  facilitated  when  the  media 
system  being  dealt  with  is  well-known  and  widely-available. 
Moreover,  for  the  production  and  reproduction  of  media  presenta¬ 
tions  for  such  standard  systems,  specialized  equipment  and 
trained  service  personnel  are  much  more  likely  to  be  available 
than  for  an  exotic  and  little-used  medium. 


2.4  Constraints  Upon  Media  Selection 

Ungoverned  by  real-world  constraints,  the  media  selection 
process  can  produce  interesting  and  effective  but  practically 
unusable  solutions.  Resource  and  personnel  availability  levels, 
costs,  and  personnel  capability  are  al'  constraints  which  exist 
in  nearly  all  media  selection  activi'^es.  Many  other  constraints 
exist  as  well  due  to  policy,  regulations,  or  limited  supply  of 
some  critical  resource.  Some  are  hord  constraints  which  cannot 
be  changed,  and  some  are  soft.  Wito  the  proper  justification 
many  can  be  modified. 

Certain  constraints  existed  at  the  onset  of  the  F-i6  media 
selection.  Some  of  these  were  due  to  the  late  timing  of  the 
contractor's  involvement  in  the  F-i  6  development  effort,  and 
others  were  due  to  general  USAi  policies  or  programs.  This  sec¬ 
tion  contains  a  study  of  the  F-16  constraints  related  to  media 
selection.  Those  listed  are  a  subset  of  the  constraints  active 
within  the  context  of  the  F-16  project.  The  total  list  of  con¬ 
straints  and  developer  response  to  them,  is  contained  in  project 
report  no.  15,  "  Prog  ram/ System  Constraints  Analysis  Report." 


2.4.1  A/B  Model  Aircraft  Mix 

Considering  the  aircraft  as  a  training  device,  the  propor¬ 
tional  mixture  of  standard  A  models  and  the  two-cockpit  B  models 
can  be  a  media  constraint.  The  ability  of  the  B  model  to  accom¬ 
modate  an  instructor  pilot  is  a  definite  advantage  for  training 
purposes  for  most  of  the  exercises  RTU  students  will  engage  in. 
The  non-availability  of  B  model  planes  for  training  purposes 
necessitates  the  use  of  two  A  model  planes:  One  for  the  student 
and  one  flying  chase.  The  net  effect  of  additional  B  models  will 
be  on  the  overall  number  of  sorties  available  for  training 
purposes.  To  the  extent  that  A  models  predominate  the  aircraft 
acquired  for  RTU  training  purposes,  a  constraint  will  be  felt. 

The  reduction  in  available  sorties  may  effect  the  media  by 
placing  a  greater  load  upon  simulation  equipment,  increasing  the 
requirement  for  such  equipment  or,  as  the  more  probable  case, 
reducing  the  amount  that  can  be  accomplished  using  available 
equipment.  Selections  for  F-16  media  must  allow  for  the  A/B 
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model  mix  in  determining  the  level  of  simulator  support  required. 
A  heavy  production  of  B  models  early  and  for  training  purposes 
should  be  planned. 


2.4.2  Highly  Volatile  Instructional  Content 

Because  F-16  is  a  highly  complex  emerging  weapon  system,  it 
can  be  expected  that  numerous  changes  will  be  made  to  the  air¬ 
craft  and  to  the  procedures  for  its  operational  use.  Changes  in 
instructional  content  are  dictated  by  these  system  and  procedural 
changes  that  in  turn  dictate  a  change  in  instructional  materials. 
Since  changes  will  be  constant,  this  indicates  a  constraint  will 
be  placed  upon  media  presentations.  It  means  they  must  be 
economical  to  revise,  not  only  in  turnaround  time  and  personnel 
time,  but  in  terms  of  special  equipment  or  technical  ability 
required  to  complete  revisions. 


2.4.3  Requirement  for  Premature  Media  Decisions 

The  difficulty  of  coordinating  instructional  development 
schedules  with  procurement  schedules  sometimes  affects  learning 
center  media  and  simulator  device  decision-making.  To  allow  the 
construction  and  equipping  of  learning  center  facilities  to 
proceed  on  an  operationally  tolerable  schedule,  the  numbers  and 
types  of  media  devices  and  the  architectural  plan  of  the  learning 
center  for  F-16's  first  training  site  had  to  be  determined  and 
announced  before  the  media  selection  process  was  executed.  These 
data  were  provided  when  requested  and  formed  the  basis  for  order¬ 
ing  learning  center  equipment  and  specifying  learning  center  lay¬ 
out.  Data  are  not  available  on  whether  the  decisions  were 
correct.  The  requirement  for  an  early  announcement,  however, 
with  the  knowledge  that  the  figures  provided  would  be  used  to 
order  media  equipment,  placed  a  constraint  upon  learning  center 
media  selections  actually  resulting.  It  is  likely  that  the  types 
of  devices  selected  in  that  early  selection  were  correct.  It  is 
not  likely,  however,  that  the  numbers  of  media  devices  was  as 
accurate  in  the  light  of  continually-revised  student  load 
figures,  actual  media  selections,  and  updates  to  critical  task 
listing  and  objectives  hierarchy  documentation. 


2.4.4  In-progress  Training  Devices 

The  coordination  of  instructional  development  decision¬ 
making  with  the  logistics  of  procuring  complex  media  devices,  for 
instance  simulator  equipment,  has  been  difficult  in  the  past  as 
mentioned  in  the  previous  section.  Often  decisions  on  such 
devices  take  place  before  the  arrival  of  instructional  syst^»ms 
development  personnel.  Alternately,  such  decisions  are  oft»»n 
made  by  developers  acting  with  inadequate  information  or  analysis 
upon  which  to  base  their  recommendation.  The  F-16  project  has 
involved  instructional  developers  at  a  very  early  phase  of  air- 
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craft  system  design  and  implementation.  Nonetheless,  the  dead¬ 
lines  for  procuring  simulators  came  earlier,  and  many  features  of 
the  F-16  OFT/WST,  CFT,  and  EPT  were  determined  in  advance  of 
developer  arrival. 

A  provision  of  the  F-16  contract  requires  input  from  the 
instructional  developer  on  the  qualities  and  capabilities  desired 
in  the  F-16  OFT.  The  formal  portion  of  this  input  is  contained 
in  project  report  no.  22,  "Recommendations  for  F-16  Operational 
Flight  Trainer  (OFT)  Improvements".  Though  it  has  been  found 
that  some  of  the  recommendations  can  be  implemented  in  the  final 
simulator  system  for  F-16,  many  of  them  cannot  because  of  the 
advanced  stage  reached  in  planning  when  the  recommendations  were 
produced.  To  the  extent  that  the  design  of  the  simulator  cannot 
be  flexible  to  the  developer's  plans,  a  constraint  is  encountered 
which  the  developer  must  circumvent  by  adapting  the  instructional 
technique  to  the  existing  trainer  capabilities. 


2.4.5  Late  Arrival  of  Ordered  Training  Devices 

Delivery  deadlines  for  already-ordered  training  devices  fall 
after  the  beginning  of  training  of  F-16  pilots.  Availability  of 
the  OFT  is  likely  to  be  four  to  six  months  following  the  start  of 
training.  Availability  of  the  first  CFT  is  scheduled  for  January 
1979.  The  Egress  Procedures  Trainer  is  to  be  installed  and 
working  by  January  1979.  The  interim  training  course  will  begin 
in  January  1979. 

Interim  simulation  capabilities  of  various  types  have  been 
proposed  by  F-16  agencies.  Each  has  attached  to  it  a  set  of 
constraints  which  will  have  major  impact  on  F-16  training,  parti¬ 
cularly  in  the  area  of  the  syllabus  sequence.  Simulation  capa¬ 
bilities  operating  at  a  distance  from  the  central  training  site 
require  extracts  of  the  syllabus  to  be  concentrated  together  for 
student  work  as  the  simulator  becomes  available.  Moreover, 
instructors  working  at  a  distance  from  the  main  training  site  are 
likely  to  encounter  communication  problems  with  the  main  site. 
Students  may  also  encounter  the  need  for  refresher  instruction, 
depending  upon  the  lag  in  simulator  availability.  Under  these 
circumstances  special  configurations  of  the  instructional  materi¬ 
als,  syllabus,  and  the  instructor  training  and  monitoring  system 
may  be  called  for. 

Simulation  capabilities  operating  at  the  main  site  but  at  a 
lesser  degree  of  sophistication  also  require  special  interim 
syllabi  and  provision  for  later  update  of  student  capability  as 
the  full-capability  simulator  becomes  available. 


2.4.6  Variable  Availability  of  Sophisticated  Media  Across 
Training  Sites 

The  existence  of  sophisticated  air  combat  monitoring  and 
evaluation  devices  such  as  ACMR/I  at  some  F-16  training  sites  and 
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their  lack  at  others  is  almost  certain.  A  variety  of  devices  now 
exist  at  various  locations,  are  scheduled  to  be  installed,  or  are 
in  various  stages  of  development  or  procurement.  The  variability 
of  these  devices  from  site  to  site  is  a  constraint  upon  media 
selection  in  the  sites  where  less  sophisticated  systems  exist. 

The  F-16  training  system  will  present  unique  opportunities  for 
evaluating  the  relative  effectiveness  of  several  of  these 
systems.  It  is  recommended  that  evaluation  be  conducted  and  the 
results  used  to  equalize  the  most  effective  of  these  systems 
across  training  sites. 


2.4.7  Absence  of  Funding  for  Additional  Equipment 

Despite  the  presence  of  previously-made  media  decisions,  the 
possibility  that  media  studies  during  instructional  development 
might  reveal  additional  desirable  training  devices  was  not  ruled 
out.  Such  devices  might  include  extra  copies  of  already-ordered 
devices,  modified  already-ordered  devices,  or  totally  new 
devices.  Should  any  of  these  prove  desirable  and  funding  for 
them  not  be  found,  that  lack  of  funding  would  require  revision  of 
media  and  syllabus  plans  and  would  constitute  a  limitation.  To 
the  extent  that  recommendations  for  changes  and  additions  to  the 
training  device  inventory  are  couched  in  cost  and  effectiveness 
terms,  a  decision  should  be  possible  on  the  desirability  of 
modifying  the  cost  constraints. 


2.4.8  Available  Media  Production  Facilities 

The  ready  access  of  some  types  of  media  production  capabil¬ 
ity  and  the  difficulty  of  obtaining  expertise  and  equipment  for 
other  types  consitutes  a  constraint  upon  certain  F-16  media 
selections.  The  instructional  materials  created  for  F-16  must  be 
maintained,  updated,  and  replaced  as  required.  Choosing  media 
for  which  production  facilities  and  personnel  were  difficult  to 
obtain  or  train  would  be  wasteful. 


2.4.9  Variable  Availability  of  Operational  Air  Space  Across 
Training  Sites 

In  the  broadest  perspective,  even  the  air  space  available 
for  practice  purposes  must  be  considered  a  part  of  media  deci¬ 
sions.  Since  the  ultimate  task  of  the  developer  is  to  place  the 
student  in  a  real-world  operational  environment  as  closely  as 
possible  for  practice  purposes,  limitations  in  air  space  (a  part 
of  the  real-world  operational  environment)  constitute  a  limita¬ 
tion  on  the  developer's  ability  to  provide  the  student  with  a 
mediated  experience.  The  climate  at  potential  training  sites 
limits  to  some  extent  the  availability  of  air  space.  These  limi¬ 
tations  are  more  than  trivial  and  have  both  cost-  and  capability- 
limiting  effects  upon  the  instructional  system. 
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3.0  MEDIA  SELECTION  PROCESS 


This  section  contains  a  step-by-step  description  of  the 
media  selection  model  used  for  F-16.  The  model  contains  a  set  of 
procedures  along  with  the  documentation  aids  necessary  to  support 
the  decision-making  process  and  record  decisions  made.  The 
description  which  follows  describes  each  step  in  the  process 
separately  at  a  level  of  detail  that  should  enable  the  media 
selection  process  to  be  performed  by  other  than  those  originating 
the  process. 

A  main  feature  of  the  model  is  its  division  into  two  main 
activities  (1)  selection  of  academic  or  learning  center  media, 
and  (2)  creation  of  specifications  for  simulator-type  training 
devices  through  a  generative  process  using  the  inventory  of  tasks 
to  be  trained  as  an  input.  The  two  areas  of  media  selection  are 
treated  as  separate  processes  because  of  the  different  approach 
which  is  required  for  selections  to  be  made  in  each  area.  For 
the  selection  of  learning  center  media,  decisions  are  made  rela¬ 
tive  to  an  existing  set  of  media  device  types  which  can  either  be 
used  as  is  or  reconfigured  to  suit  strategy  requirements.  These 
devices  are  oriented  toward  the  imparting  of  information,  the 
formation  of  new  concepts,  the  learning  and  practice  of  new 
rules,  and  other  cognitive  exercises.  The  amounts  of  engineering 
involved  in  their  selection  is  minimal. 

On  the  other  hand,  training  devices  which  are  not  of  this 
group,  are  not  prepared  mainly  for  the  establishment  of  the  con¬ 
ceptual  and  informational  base  but  for  the  practice  of  actual  job 
behaviors  or  behavior  fragments  in  simulated  job  environments. 
These  devices  may  be  selected  from  a  predetermined  set  of  catego¬ 
ries  but  must  also  be  engineered  individually  to  create  this  job- 
related  environment.  Off-the-shelf,  unmodified  media  devices  are 
not  available  ready-made  to  meet  a  particular  training  need,  and 
"design"  of  devices  may  be  a  more  appropriate  term  than  "selec¬ 
tion"  for  the  process  involved  in  determining  what  is  needed  for 
the  training  system.  The  training  device  design  model  presented 
in  this  report  attempts  to  specify  the  best  family  of  training 
devices  for  a  given  set  of  tasks  to  be  trained,  each  engineered 
for  its  particular  training  function. 

The  general  limitations  in  media  selection  methodology 
described  in  Section  1.0  of  this  report  apply  to  both  the  learn¬ 
ing  center  media  selection  and  the  training  device  design  proc¬ 
esses.  At  present  the  questions  of  media  selection  have  no  firm 
answers.  For  the  selection  of  learning  center  media,  few  firm 
guidelines  are  provided  to  developers  other  than  the  rationale 
each  one  can  establish.  For  the  selection  of  simulator-type 
training  devices,  a  body  of  coherent,  consistent  procedures  m^ant 
to  derive  training  device  specifications  from  statements  of  job 
task  requirements  has  begun  to  evolve,  but  the  procedures  arp  by 
no  means  complete,  nor  has  the  decision-making  path  and  factor 
balancing  mechanism  the  designer  must  follow  as  specifications 
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for  such  devices  are  derived  been  sufficiently  discussed  as  an 
entity.  The  methodology  in  this  report  is  offered  as  a  best 
attempt  at  establishing  a  procedure.  Much  research  and  empirical 
testing  must  determine  changes  to  it  which  will  lead  to  better 
selection  and  designs. 

Inherent  in  the  procedures  described  in  this  report  are 
certain  assumptions: 

(1 )  Learning  center  media  should  be  selected  with  due  pro¬ 
portional  emphasis  upon  the  instructional  strategy  to  be  used 
during  instruction.  Several  media  selection  models  acknowledge 
this  principle  and  attempt  to  define  the  elements  of  instruc¬ 
tional  strategy  which  figure  in  media  decisions.  For  the 
purposes  of  this  report  a  perspective  of  instructional  strategy 
is  taken  which  examines  the  individual  instructional  displays  to 
be  received  by  the  student,  classifies  them,  and  utilizes  system¬ 
atic  rules  for  their  delivery  to  the  student.  This  assumption  is 
necessary  to  avoid  the  evils  of  a  cost-factor  s-onl  y  media  selec¬ 
tion  model  and  to  give  balance  to  media  selection  by  considering 
more  of  the  relevant  factors  of  media  selection. 

(2)  Alternate  media  selections  produced  by  this  model  are 
assumed  to  be  equivalent  in  instructional  effectiveness.  This  is 
a  difficult  assumption  which  virtually  all  developers  reject  on 
the  basis  of  intuition.  Given  the  present  state  of  the  art  in 
measuring  instructional  effectiveness,  however,  and  the  inability 
of  instructional  psychologists  to  produce  any  scale  measuring 
units  of  effectiveness  for  comparison  purposes,  this  assumption 
is  necessary.  All  media  stlection  models  which  take  cost  into 
account  must  make  this  assui..ption  or  else  account  for  variable 
effectiveness  through  an  estimation  method. 

(3)  Training  device  specifications  can  be  generated  from 
statements  of  job  task  requirements  using  a  systematic  procedure. 
If  this  is  not  possible,  instructional  developers  and  simulator 
builders  must  rely  forever  on  best  guesses  and  intuitive  proc¬ 
esses  to  produce  trainer  designs  and  the  trainer  consumer  must  be 
resigned  to  the  unanswered  question  of  whether  or  not  the  lowest 
training  cost  is  being  paid. 

(4)  The  learning  center  and  training  device  selection  and 
design  process  can  be  improved  through  a  process  of  validation 
which  compares  intended  with  actual  outcomes  in  student  perform¬ 
ance.  Though  this  sounds  reasonable  and  desirable,  the  requisite 
research  and  real-world  effectiveness  studies  have  not  been  per¬ 
formed  which  supply  the  necessary  data  base.  Cost  figures  and 
effectiveness  indices  are  required  which  can  be  used  as  inputs  to 
the  selection  and  design  process  and  relieve  the  best-guessing 
which  otherwise  must  take  place.  The  ability  to  select  and 
design  media  through  a  process  which  involves  considerations  of 
cost  and  effectiveness  requires  either  a  beginning  data  base  from 
which  the  figures  can  be  derived  or  else  an  initial  set  of 
estimates  which  can  be  validated  through  experience.  That  data 
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base  does  not  presently  exist  for  aircrew  training  devices  and 
must  be  built  before  more  precise  trade-off  studies  can  be  made. 
This  is  emphasized  as  a  critically  important  point  if  progress  is 
to  be  made  in  defining  the  technology  of  media  device  design  and 
selection.  The  ability  to  do  this  further  assumes:  (i)  The 

existence  of  an  adequate  performance  measurement  and  record  keep¬ 
ing  system  within  training  systems  which  is  capable  of  collecting 
student  performance  data  of  fine  enough  detail  and  with  suffi¬ 
cient  reliability  to  produce  dependable  conclusions,  and  (2)  that 
a  data  recording  system  is  in  place  on  a  continuous  basis  as 
training  devices  are  used  to  gather  time  and  usage  data  to  be 
used  in  conjunction  with  the  performance  data. 

Figure  2  contains  a  flowchart  representation  of  the  media 
selection  process.  The  description  which  follows  refers  to  the 
individual  steps  in  this  chart.  The  initial  input  to  the  media 
selection  process  is  a  complete  list  of  c r i te r ion- r e f e r enced 
objectives  (CROs)  and  instructional  objectives  from  objectives 
hierarchy  analysis.  Objectives  are  grouped  together  during 
syllabus  building  for  combined  mediation  as  instructional  "seg¬ 
ments".  For  F-lfi  these  groups  will  not  average  over  two  to  three 
objectives  per  segment,  and  the  types  of  objective  (and  thus  the 
type  of  instructional  strategy  required  to  teach  it)  will  not  be 
mixed  within  segments  where  it  can  be  avoided.  Though  the  word 
"objective"  is  used  in  this  report  to  designate  the  instructional 
event  to  which  a  medium  will  be  assigned,  it  should  be  understood 
that  refers  to  an  instructional  segment.  These  segments  make  up 
the  inventory  of  individual  instructional  events  which  must  be 
mediated.  The  derivation  of  CROs  and  a  description  of  their 
purpose  is  contained  in  F-16  Aircrew  Training  Development  Project 
Report  No.  5,  "Derivations,  Formatting,  and  Use  of  Criterion- 
referenced  Objectives  (CROs)  and  Cr i ter i on- r e f er enced  Tests 
(CRTs)".  The  same  description  for  objectives  hierarchy  analysis 
is  contained  in  project  report  no.  8,  "Ojectives  Hierarchy 
Analysis  Methodology  Report". 

A  final  note  on  bookkeeping  requirements  and  the  handling  of 
changes  is  in  order.  The  execution  of  many  ISD  processes, 
particularly  in  large-scale  instructional  development  where  much 
information  and  data  are  generated,  the  bookkeeping  process 
required  becomes  extremely  important.  It  is  unavoidable  for  the 
developer  to  become  disoriented  in  the  midst  of  a  disorganized 
pile  of  data  which  he  generates,  but  with  an  adequate  bookkeeping 
process  it  is  easy  to  make  large  numbers  of  media  decisions 
systematically.  It  is  also  easy  given  appropriate  bookkeeping 
practices  to  insert  changes  to  the  body  of  the  data  base  created 
by  media  selection.  On  the  other  hand,  without  such  systematic 
procedures,  especially  for  bookkeeping  on  decisions  made,  change 
made  can  result  in  confusion,  duplication  of  effort,  loss  of 
data,  misplaced  training  requirements,  and  eventually  produce 
wasteful  results.  The  media  selection  process  described  in  this 
document  depends  upon  a  bookkeeping  system  of  forms  and  tables 
which  collect  and  summarize  data  for  individual  objectives  during 
the  media  selection  process.  These  preserve  a  data  trail  for 
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review  and  use  by  those  charged  with  revision  responsibilities. 
There  is  nothing  sacred  about  the  system  of  forms  beyond  the  fact 
that  they  represent  a  bookkeeping  system  which  is  adequate  to 
record  the  decisions  made  without  losing  them-  Alternate  methods 
of  bookkeeping,  particularly  computerization  of  the  process,  are 
acceptable  and  desirable  to  the  extent  that  they  minimize  the 
bookkeeping  effort  without  decreasing  the  developer's  ability  to 
execute  the  process  and  keep  from  losing  his  orientation  within 
the  vast  process  of  media  selection. 


3.1  Step  1:  Sort  Objectives  into  Academic  and  Training  Device 
Groups 

During  this  step  CROs  and  objectives  are  sorted  into  two 
classes.  The  first  class  is  made  up  of  all  those  objectives 
whose  behavior  requires  real  or  simulated  job  environments  or 
equipment.  This  is  the  "training  device"  group  of  objectives. 

The  second  class  contains  objectives  whose  behaviors  can  be 
performed  without  such  equipment  or  environments.  This  is  the 
"academic"  group  of  objectives. 

There  are  two  keys  to  making  this  discrimination.  The  first 
is  the  exact  wording  of  the  behavior-  If  the  behavior  reads 
"perform  the  pre-takeoff  inspection,"  then  the  behavior  requires 
equipment  to  be  inspected.  If  on  the  other  hand  the  behavior 
reads  "list  the  items  to  be  inspected  in  the  pre-takeoff  inspec¬ 
tion"  no  equipment  is  indicated,  and  the  behavior  can  be  accomp¬ 
lished  outside  of  a  real  or  simulated  job  setting. 

Conditions  and  standards  of  an  objective  or  CRO  are  the 
second  key  to  making  the  discrimination.  A  condition  may  specify 
the  type  of  display  to  be  presented  to  the  student.  For 
instance,  a  condition  may  require  the  student  to  identify  certain 
items  from  a  radar  return  "given  a  photograph  of  a  return".  In 
this  case  the  type  of  performance  environment  is  specified.  In 
the  absence  of  such  direction,  a  real-world  environment  should  be 
assumed . 

For  those  objectives  requiring  equipment.  Steps  15  to  31  of 
the  media  selection  process  are  applied  next.  For  all  others. 
Steps  2  to  14  are  used.  Worksheets  to  facilitate  the  bookkeeping 
chores  of  media  selection  are  set  up  at  this  time.  For  each 
academic  objective  the  appropriate  numerical  entry  is  made  in 
Table  1,  the  academic  objective  analysis  worksheet.  Later  steps 
will  add  data  to  this  worksheet,  which  is  only  set  up  during  this 
step.  For  each  training  device  objective,  a  similar  table  will 
be  generated  as  described  in  Steps  15  to  31. 


3.2  Step  2:  Classify  Academic  Objectives 

Academic  objectives  are  classified  during  this  step  accord¬ 
ing  to  a  predefined  scheme  of  categories.  The  categories  are  an 
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integral  part  of  tho  objective's  hierarchy  analysis  process,  in 
that  analysis  normally  takes  place  with  a  given  set  of  objective 
types  in  the  developer's  mind  as  likely  outcomes  of  analysis,  and 
the  categories  of  objectives  to  be  produced  through  analysis  are 
usually  determined  prior  to  analysis.  If  this  is  not  the  case, 
they  may  be  classified  in  retrospect  following  it. 

The  classification  of  objectives  at  this  point  prepares  the 
way  for  the  assignment  of  instructional  strategtf^s.  In  subse¬ 
quent  steps  of  the  media  selection  process,  instructional  strat¬ 
egics  will  be  specified  for  each  class  of  instructional  objec¬ 
tives,  with  the  assumption  being  that,  a  generic  strategy  pattern 
will  be  used  (with  local  adjustments  for  content  characteristics) 
for  a  given  class  of  objectives.  When  the  specific  properties  of 
the  generic  strategies  are  designed,  the  operational  characteris¬ 
tics  of  media  devices  able  to  present  those  strategies  will  be 
defined  in  consequence. 

For  the  F-16  project,  academic  objectives  are  classified 
according  to  the  following  types  (see  Attachment  I  for  further 
elaboration  and  examples): 

Remember/Pr ocedure  ( ver bat im) --these  objectives  require 
students  to  be  able  to  recite,  verbatim, 
procedures  for  operating  the  f-16.  (RPV) 

Remember/Procedure  (paraphrase) — these  objectives 

require  the  student  to  name  only  critical  steps 
and  critical  values  during  a  procedure.  Emphasis 
is  on  paraphrased  recall  or  on  recognition  of  a 
correct  answer  when  it  is  seen.  (RPP) 

Remember/Fact  (symbol/name) — these  objectives  require 
the  association  between  a  symbol  and  a  name  or 
between  a  symbol  and  a  definition.  (RFSN) 

Remember /Fact  ( funct ion/locat ion) --these  objectives 
require  the  student  to  associate  the  name, 
function,  and  location  of  a  given  object.  (RFFL) 

Remember/Fact  (limits/tolerances/critical  values) --th i s 
objective  requires  the  student  to  learn  an 
isolated  fact  or  numerical  value.  (RFC) 

Remember/Fact  (organized  lists) — these  objectives 

require  the  student  to  learn  a  list  of  events, 
items  or  objects  which  are  in  some  way  related 
to  each  other  but  not  necessarily  in  sequence. 
(RFOL) 

Remember/Fact  (sequences) — these  objectives  require  a 

student  to  recite  a  particular  sequence  of  items 
in  correct  order.  The  sequence  must  be  one  that 
the  student  will  not  be  called  upon  to  perform. 


li 
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for  instance,  the  sequence  of  operation  for  a 
physical  system.  (RFS) 


Usp'/Concept — these  objectives  require  the  student  to 
identify  examples  and  non-examples  of  a  given 
class  of  things.  All  items  classified  must  be 
previously  unencountered.  (UC) 

Use/Rule--these  objectives  require  a  student  to  apply  a 
computational  rule  or  a  genf^ral  principl<^  in  the 
solution.  (UR) 

Once  an  objective  type  has  been  determined,  the  appropriate 
entry  should  be  made  on  the  Academic  Objective  Analysis  Worksheet 
(Table  1)  in  the  "OBJ  TYPE"  column.  Abbreviations  following  the 
objectives  descriptions  above  may  be  used. 


3.3  Step  3:  Specify  Instructional  Strategies  and  Alternates 

The  generic  strategies  are  specified  at  this  time  for  each 
class  of  instructional  objective. 

The  F-16  instructional  strategies  evolve  from  a  philosophy 
of  strategy  which  regards  instructional  presentations  as 
sequences  of  individual,  classifiable  packets  of  information 
(displays)  having  a  consistent  logical  relationship  to  each 
other.  The  family  of  displays,  interdisplay  relationships,  and 
response  requirements  which  ate  needed  in  the  teaching  of  a  given 
objective  can  be  identified.  Since  the  medium  is  the  device  for 
presenting  displays  and  handling  all  interactions  between  the 
student  and  display  contents,  the  types  of  displays  and  the 
interactions  required,  are  important  to  define. 

When  strategies  have  been  specified,  a  Strategy  Media 
Requirements  Profile  is  filled  out  for  each.  This  table  summa¬ 
rizes  display,  response,  and  other  characteristics  of  each 
strategy.  A  sampl*^  profile  is  given  in  Table  2. 


3.4  Step  4;  State  All  Nonstrategy  Factors  Affecting  Media 
Selection 

In  Step  4  non-strategy  factors  affecting  m^dia  selection  are 
listed.  These  include  many  of  the  practical  requirements  arising 
from  the  patterns  of  day-to-day  use.  They  take  into  account  the 
needs  of  students  using  the  media  and  the  needs  of  administrators 
buying  and  maintaining  the  media  devices  and  media  presentations 
used  on  them.  They  take  note  of  special  environments  for  device 
use  and  help  insure  that  devices  which  are  picked  are  suited  for 
use  in  all  required  environments.  The  factors  affecting  media 
selection  must  include  constraints  which  will  exist  as  media  are 
selected.  These  constraints  need  not  be  looked  upon  as  absolute 
barriers  unless  they  have  unequivocally  been  identified  as  that. 
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Strategy  Media  Requirements  Profile 
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Table  2  -  Strategy  Media  Requirements  Profile 


It  is  possible  that  constraints  may  yield  to  a  sound  media 
rationale . 

It  is  important  to  identify  these  factors  in  advance  of 
media  selection.  When  the  factors  are  identified  they  should 
also  be  prioritized  or  given  proportional  weight  as  to  their 
relative  importance.  These  weights  are  used  to  balance  the 
factors  to  ensure  that  more  than  one  factor  is  considered  and 
that  undue  emphasis  is  not  placed  on  one  factor.  A  process  for 
externalizing  these  priorities  is  not  offered  here,  for  it  is  a 
complex  judgement  process,  balancing  factors  agains  each  other. 
The  main  factors  influencing  media  selection  for  F-16  have  been 
presented  in  Section  2.3  and  2.4  of  this  report. 


3.5  Step  5:  List  the  Initial  Media  Pool 

During  this  step  a  list  is  made  of  all  the  devices  which 
will  be  candidates  during  media  selection.  All  individual  media 
types  and  combinations  of  types  which  are  thought  to  be  usable 
should  appear  on  this  list.  In  subsequent  steps  entries  will  be 
deleted  from  this  list  which  (a)  do  not  effectively  implement 
instructional  strategies  and  (b)  do  not  support  the  factors 
listed  in  Step  4. 


3.6  Step  6:  Eliminate  and  Reconfigure  Media  to  Produce  the 
Acceptable  Media  List 

Once  the  strategy  and  non-strategy  factors  have  been  stated 
which  will  limit  or  constrain  media  selection,  the  master  list  of 
all  available  media  can  be  scanned  for  media  which  do  not  support 
the  factors,  or  which  are  without  the  bounds  of  the  constraints 
and  for  which  there  seems  no  possibility  of  removing  them.  These 
media  are  deleted  from  the  candidate  media  list.  Some  possible 
reasons  for  eliminating  a  medium  might  be; 

-  The  immediate  or  long-range  costs  associated  with  a  medium 
for  production,  repair,  acquisition,  or  operation  are 
prohibitively  high  when  weighed  against  the  expected 
benefits . 

-  The  expertise  or  equipment  required  to  produce  and  modify 
instructional  presentations  is  not  readily  available. 

-  The  medium  does  not  fall  within  the  limits  set  by  the 
constraints  (see  Step  4) . 

-  The  medium  is  as  yet  experimental  in  nature  and  has 
unknown  operational  properties  (durability,  student 
acceptance,  etc). 

-  The  production  time  for  presentations  in  this  medium  is 
too  long. 
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-  The  medium  is  not  durable  and  is  subject  to  breakdowns. 

-  The  medium  makes  demands  of  students  that  are  beyond  their 
normal  range  of  interest  or  ability. 

-  The  medium  is  too  complex  to  use  and  is  likely  to  require 
too  much  of  the  student's  time  or  attention  for  operation. 

These  are  suggested  media  elimination  criteria  and  do  not 
comprise  an  exhaustive  list.  A  complete  list  for  a  given  media 
selection  will  depend  upon  factors  local  to  the  project. 

A  Media  Elimination  Table  like  that  in  Table  3  is  filled  out 
as  a  record  of  the  rationale  for  eliminating  a  given  type  of 
media.  This  captures  the  logic  of  the  developer  for  future  re- 
evaluation  of  decisions.  Only  those  media  remaining  on  the  list 
after  this  elimination  and  a  screening  for  strategy  capability 
will  be  considered  for  selection. 

In  making  deletions  some  judgement  is  required.  Decisions 
are  not  likely  to  be  clear-cut,  and  some  trade-offs  may  be 
considered  before  eliminating  media.  For  instance,  some  media 
may  be  clearly  unaffordable,  too  delicate,  not  portable,  or  in 
some  other  way  beyond  practicality.  Other  media,  on  the  other 
hand,  may  appear  to  be  out  of  bounds  on  some  factors  but  yet  have 
such  a  strong  positive  recommendation  from  others  that  they  may 
be  retained  on  the  list.  A  medium  which  is  expensive  in  acquisi¬ 
tion  but  yet  can  implement  instructional  strategies  far  better 
than  other  media  may  be  retained  as  a  candidate.  A  medium  which 
has  found  itself  in  this  situation  in  the  past  is  computer- 
assisted  instruction  (CAI)  .  The  initial  cost  of  a  CAI  system  is 
often  troublesome,  but  the  attraction  of  the  strategy  versatility 
of  a  computer  and  the  side  uses  of  a  computer,  for  instance,  in 
computer-managed  instruction  (CMI),  make  a  CAI  system  very  desir¬ 
able  despite  cost.  (Surprisingly,  when  total  actual  costs  are 
compared  for  various  media  some  CAI  systems  turn  out  to  be 
comparatively  less  expensive.  This  is  because  actual  costs  are 
in  fact  higher  than  usually  estimated  and  CAI  costs  lower  than 
expected.)  This  combination  could  cause  a  medium  to  be  left  on 
the  candidate  list  despite  some  factors  legislating  against  it. 
This  has  been  the  case  for  F-16.  When  media  are  for  some  reason 
desirable  but  cannot  fit  the  requirements  imposed  by  the  factors, 
reconfiguration  of  a  medium  by  combination  with  another  medium  or 
by  different  patterns  of  use  should  be  considered.  Two  media 
used  in  combination  may  be  more  effective  than  a  single  medium. 

Instructional  strategies  are  considered  at  this  time.  Media 
are  eliminated  from  the  candidate  list  which  are  unable  to  imple¬ 
ment  the  display,  response,  and  other  requirements  of  at  least 
one  strategy.  At  the  bottom  of  the  Strategy  Media  Requirements 
Profile  pages  already  filled  out  in  Step  3  (see  Table  2)  those 
media  capable  of  implementing  each  one  are  entered,  along  with 
any  notes  on  how  each  medium  can  be  changed  or  used  differently 
to  fill  some  or  all  requirements  of  the  strategy.  Because  some 
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strategies  may  require  complex  capabilities  not  readily  avail¬ 
able,  media  which  can  implement  significant  portions  of  a 
strategy  should  be  listed. 

The  media  which  are  listed  on  the  bottoms  of  the  profile 
pages  following  this  procedure  comprise  the  final  media  list. 


3.7  Step  7:  Oetermine  Special  Instructional  Media  Requirements 
for  Each  Objective 

The  Academic  Objective  Analysis  Worksheet  (Table  1)  lists 
specific  media  considerations  under  the  headings  of  display 
requirements,  control  and  pace  requirements,  response  require¬ 
ments,  scoring  requirements,  and  feedback  requirements.  During 
this  step,  under  each  of  these  headings,  a  mark  is  placed  next  to 
the  requirement  for  instruction  for  each  objective.  In  Step  3 
completing  the  Media  Strategy  Requirements  Profile  (Table  2)  made 
a  general  statement  of  required  media  characteristics  for  a 
generic  strategy.  Now  during  this  step,  the  additional  effect  of 
the  specific  content  of  an  objective  is  taken  into  account.  Some 
objectives  may  deal  with  content  which  requires  more  detailed 
illustrations  or  color  or  motion  in  a  way  which  departs  from  the 
general  strategy  pattern.  Using  the  media  strategy  requirements 
profile  as  a  general  guide,  the  developer  should  identify 
requirements  for  each  objective  under  each  heading.  Requirements 
which  deviate  from  the  general  pattern  should  be  noted  in  the 
rightmost  column  so  that  special  adjustment  may  be  made  at  the 
time  of  media  decision-making. 

Some  general  guidelines  may  be  provided  for  specifying  the 
requirements.  For  display  capability  choose  the  least  compli¬ 
cated  or  expensive  level  of  effort  which  will  accomplish  the 
instruction.  Unfortunately  there  is  some  subjectivity  in  this 
decision.  The  instruction  to  choose  the  least  complicated 
capability  arises  from  the  tendency  of  media  research  to  find 
little  or  no  difference  in  learning  attributable  to  more  complex 
or  finely  finished  instructional  displays  (Travers,  1970). 

Simple  line  drawings  are  in  many  cases  as  instructional  ly  effec¬ 
tive  as  complex  renditions  or  even  photographs.  There  is  even 
some  evidence  that  during  training  the  more  complex  the  display, 
the  more  difficult  it  is  for  the  student  to  detect  the  main 
message  or  salient  points. 

For  pace  and  control  decisions,  student  control  should  be 
favored,  depending  upon  the  anticipated  motivation  and  sel  f- 
directedness  of  the  students.  Research  in  military  training 
environments  has  found  time  efficiencies  attendant  to  students 
being  given  control  of  instructional  order  and  speed  (Walker, 
1978).  For  students  who  are  highly  motivated  and  self-directed 
this  effort  would  be  expected  to  be  very  strong.  On  the  other 
hand,  for  dealing  with  low-motivation  students  or  students  who 
are  not  self-directed,  student  control  may  be  inappropr  .ate . 
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3.8  Step  8;  Determine  Alternate  Media  Assignment  Patterns 


As  non-strategy  factors  affecting  media  selection  (e.g., 
cost,  time  of  revision,  production  facilities  required  for 
revision)  change,  the  desirability  for  different  media  assignment 
patterns  also  changes.  When  videotape  production  facilities  are 
readily  available  and  relatively  inexpensive,  the  desirability  of 
using  videotape  as  a  medium  increases.  When  the  capability  does 
not  exist  or  ceases  to  exist,  the  desirability  of  that  mixture 
diminishes,  and  the  media  assignment  pattern  will  specify  a  lower 
percentage  of  videotape  instruction.  This  step  involves  the 
specification  of  desirable  patterns  of  media  assignment  growing 
out  of  priorities  and  constraints  found  in  the  media  selection 
factors  - 

A  variety  of  priority  orderings  of  media  ate  possible  given 
different  priorities  in  the  factors  which  are  operating  on  media 
selection.  The  developer  may  create  several  media  assignment 
patterns  prior  to  media  selection  in  hopes  of  finding  the  best 
balance  of  cost,  strategy,  and  other  selection  factors.  The 
creation  of  patterns  for  this  testing  and  fitting  process  entails 
either  (1)  the  ordering  of  the  media  listed  on  the  Strategy  Media 
Requirements  Profile  (Table  2)  in  order  of  preference  according 
to  a  particular  factor  balance,  or  (2)  the  setting  of  specific 
ceiling  percentages  for  specific  media  on  that  list  beyond  which 
assignments  will  not  be  allowed  to  go.  A  developer  may  wish  to 
favor  media  which  can  implement  specific  instructional  strategies 
despite  their  cost  and  will  arrange  media  on  all  the  profiles  in 
that  ordering.  At  the  same  time  the  developer  may  also  wish  to 
see  the  effects  of  taking  cost  into  account  as  the  main  factor 
and  so  will  order  the  media  in  a  second  assignment  pattern  in 
order  of  cost.  In  a  third  pattern  the  developer  may  wish  to  test 
the  effects  of  stipulating  that  tape/slide  presentations  will  not 
take  up  more  than  30%  of  the  total  media  assignments. 

Table  4  contains  a  form  for  the  purpose  of  writing  down 
media  assignment  patterns.  One  form  is  completed  for  each 
pattern  created.  All  media  still  in  consideration  are  gathered 
from  the  bottoms  of  the  Strategy  Media  Requirements  Profiles 
(Table  2)  and  entered  in  the  left-hand  column.  Each  objective 
type  is  entered  at  the  head  of  its  own  column.  For  each  objec¬ 
tive  type  an  entry  is  made  after  the  media  to  be  used  with  that 
type  of  num.-rical  priority  (1st  choice,  2nd  choice,  etc.)  of 
each.  Ceiling  percentages  are  entered  where  that  is  a  priority 
factor.  As  well  as  keeping  a  record  of  the  patterns  of  interest, 
a  record  should  be  made  for  each  pattern  of  the  benefits  and 
drawbacks  associated  with  each.  The  back  of  the  record  form  is 
used  for  this.  This  record  will  become  an  important  aid  to 
decision-making  in  the  final  balancing  of  media  factors  in  Step 
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MEDIA  ASSIGNMENT 
PATTERN  RECORD  FORM 


Table  4--Media  assignment  pattern  record  form. 


3.9  Step  9:  Deteraine  Media  Production  and  Maintenance  Costs 

The  costs  attending  the  use  of  each  medium  must  be  known. 
Thses  costs  include  developnent  and  operation  costs. 


3.10  Step  10:  Construct  the  Media  Decision  Models 

Media  selection  decision  models  are  constructed,  one  for  each 
media  assignment  pattern.  The  factors  balanced  in  this  model 
include  instructional  strategy,  cost,  and  any  other  factors  the 
developer  has  wished  to  include  in  media  assignment  patterns. 

Media  suitable  for  each  strategy  type  have  been  chosen  in  earlier 
steps  of  the  selection  process,  and  various  priority  orderings  have 
been  created  to  test  the  effects  of  different  developer  decisions 
on  media  mixtures.  The  media  selection  model  is  the  specific  rule 
for  assigning  media  to  objectives  and  takes  into  account  all  of 
these  earlier  decisions. 

A  media  selection  decision  model  may  be  represented  as  a 
sequence  of  decision  points  which  executes  the  following:  (1) 

identification  of  the  objective  by  type,  (2)  matching  of  specific 
instructional  content  demands  to  display,  response,  and  other  media 
characteristics  for  each  objective,  (3)  filtering  of  the  total 
number  of  objectives  assigned  to  one  medium  if  percentages  have 
been  specified  as  part  of  assignment  patterns,  and  (4)  assignment 
of  a  specific  medium. 

Each  assignment  pattern  created  during  Step  8  requires  a 
separate  model,  since  the  order  ih  which  media  are  preferred  is 
different  in  each  one. 


3.11  Step  11:  Select  Media  for  All  Objectives  Under  Each 
Assignment  Pattern 

At  this  point  media  selection  proceeds  for  each  objective 
under  each  assignment  pattern.  Selection  of  media  for  each  media 
plan  takes  place  separately.  Each  objective  is  passed  through  the 
logic  of  each  assignment  pattern  decision  model  and  has  a  medium 
assigned  to  it.  If  there  are  four  assignment  patterns  used,  each 
objective  must  be  passed  through  four  models.  Computerization  of 
this  process  may  be  desirable  for  large  volumes  of  objectives. 

A  record  of  media  selections  is  kept  for  each  objective  under 
each  pattern.  Table  5  contains  the  table  used  for  this  purpose. 

It  may  take  the  form  of  an  overlay  to  Table  1  to  eliminate  duplica¬ 
tion  of  records. 

A  summary  is  made  of  media  selections  once  they  have  been  made 
and  recorded.  Table  6  contains  a  summary  table  used  to  report  the 
number  of  objectives  selected  for  each  medium  under  each  assignment 
pattern  . 


30 


MEDIA  SELECTION  RECORD  TABLE 


'^1 


Table  5 — Media  selection  record  form. 


Media  Assignment  Pattern 


For  media  whose  data  on  the  Academic  Objective  Analysis 
Worksheet  (Table  1)  showed  a  divergency  from  the  general  strategy 
pattern  prescribed  for  its  objective  type,  adjustments  in  selec¬ 
tions  are  made  on  an  individual  basis  as  needed. 


3.12  Step  12;  Determine  the  Total  Cost  of  Each  Media  Assignment 
Pattern 

Using  the  cost  figures  for  each  medium  derived  in  Step  9, 
the  cost  of  each  media  mixture  derived  through  the  assignment 
process  in  Step  11  is  determined.  These  costs  may  not  be  sensi¬ 
tive  to  variations  in  volume,  and  adjustments  should  be  made  for 
a  higher  or  lower  volume  of  production  than  was  used  to  derive 
the  original  cost  figures. 


3.13  Step  13:  Review  Cost  Tradeoffs  Related  to  Each  Assignment 
Pattern 

The  positive  and  negative  factors  listed  for  each  media 
assignment  pattern  in  Step  8  and  attached  to  it  are  now  examined 
along  with  the  costs  derived  in  Step  12.  It  is  possible  to  con¬ 
duct  an  informal  cost  tradeoff  study  at  this  time.  The  developer 
should  choose  that  media  configuration  for  recommendation  which 
presents  the  best  set  of  positive  instructional,  cost  and  use 
features.  No  set  of  guidelines  should  be  set  down  for  this 
because  it  involves  a  decision-making  process  on  the  part  of  the 
developer  and  the  instructional  system  consumer  which  may  involve 
many  external  factors  to  the  system  itself. 

By  producing  the  media  assignments  and  the  cost  data 
attendant  to  each  for  several  assignment  patterns  the  developer 
has  laid  the  foundation  for  orderly  decision-making  and  compar¬ 
ison  of  several  ways  to  proceed.  Further  assignment  patterns  may 
be  generated  at  this  point  to  test  new  possibilities  for  media 
mixtures,  and  it  is  assumed  that  a  negotiation  between  the  devel¬ 
oper/consumer  and  the  real-world  contingencies  will  take  place  at 
this  time  until  the  best  combination  of  media  for  the  funds 
available  is  determined.  Moreover,  when  the  decision  is  reached, 
both  developer  and  consumer  will  have  an  understanding  of  the 
capabilities  and  limitations  of  the  resulting  media  system. 


3.14  Step  14:  Summarize  Media  Resource  Requirements 

A  summary  should  be  made  containing  the  numbers  and  types  of 
media  devices  to  be  acquired  along  with  list  of  the  media  produc¬ 
tion  facilities  required  to  support  the  production  and  revision 
of  media. 
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3.15  Step  15;  Analyze  Behavior  Elements  Contributing  to  Task 
Difficulty 

In  the  performance  of  any  task,  there  are  certain  things 
which  may  make  the  task  difficult.  The  difficulty  may  arise  from 
any  number  of  causes;  the  complexity  of  task  integration,  heavy 
drains  on  memory,  complex  and/or  precise  motor  performances  which 
are  part  of  the  task,  difficult  discriminations,  complex  calcula¬ 
tions  or  decision  making  algorithms,  noise  or  interference  in  th^ 
task  environment,  timing  constraints,  or  other  task  factors. 

During  this  step  those  task  elements  which  contribute  to 
task  difficulty  are  identified  and  isolated.  Two  general  types 
of  factors  are  considered:  (1)  specific  difficult  within-task 
elements,  and  (2)  general  condition  of  task  execution. 

Difficult  within-task  elements.  Pilot  tasks  are  struc¬ 
turally  quite  similar.  They  usually  consist  of  much  the  same 
behaviors  repeated  in  a  cyclic  fashion.  A  generic  pilot  task  can 
be  specified  containing  these  seven  or  eight  common  beh.^viors. 
Then  task  difficulty  can  be  shown  primary  cases  to  be  attribut¬ 
able  to  a  difficult  sub-element  of  the  tasks.  For  instance, 
"operate  controls"  is  a  behavior  common  to  most  aircrew  tasks. 
Control  operations  for  some  tasks  requires  only  th<^  pressing  of  a 
button  or  throwing  of  a  switch.  For  these  tasks,  task  difficulty 
is  probably  not  because  of  the  difficulty  of  the  control  opera¬ 
tion.  For  other  tasks,  however,  the  required  operation  of  the 
controls  may  require  split-second  timing  and  high  precision,  or 
it  may  require  precise  movements.  The  contribution  of  control 
operation  to  the  difficulty  of  this  task  is  far  greater. 

Major  subdivisions  of  the  generic  task  for  F-16  are;  (1) 
reading  of  interior  cockpit  indications,  (2)  reading  of  exterior 
cockpit  indications,  (3)  determining  of  the  next  course  of 
action,  (4)  locating  controls,  and  (5)  operating  controls. 

General  conditions  of  task  executions.  Conditions  during 
task  execution  may  also  make  tasks  difficult.  A  general  condi¬ 
tion  of  stress  during  execution,  the  requirement  for  precise 
timing,  interfering  noise,  and  the  requirement  to  act  without 
looking  are  conditions  which  may  make  a  task  difficult. 

The  analysis  of  task  element  difficulty  factors  produces  a 
table  like  that  in  Table  7,  a  Task  Element  Difficulty  Rating 
Sheet.  The  factors  shown  on  this  table  are  those  which  appear  to 
influence  difficulty  of  F-16  tasks  most  regularly.  This  list  was 
determined  by  a  walk-through  of  representative  F-16  tasks.  Not 
all  task  elements  appearing  on  the  difficulty  rating  sheet  are 
part  of  every  F-16  task,  but  it  is  expected  that  for  the  majority 
of  F-16  tasks  the  elements  contributing  to  their  difficulty  will 
be  found  in  this  list.  If  a  task  is  encountered  which  has 
unusual  difficulty  elements,  these  elements  should  be  added  to 
the  list. 
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TASK  ELEMENT  DIFFICULTY  RATING  SHEET 


Task  or  objective  No. 

Analyst  _  Date _  Notes 


Read  External  Indicator  (Cockpit  Ext.) 
Perceive  alarm 
Perceive  object  (isolated) 

Search  and  identify  [PRE]  _ 

Locate  and  track  [PRE]  _ 

Read  Internal  Indicator 
Perceive  alarm 

Perceive  subtle  indicator  change 
Read  Discrete  or  binary  valve 
Read  continuous  scale 
Classify  element  of  a  display  [PRE] 

Determine  Course  of  Action 
Retrieve  from  memory 
Simple  calculation 

Complex  calculate  [PRE]  _ 

Complex  integrate  [PRE] 

Locate  Controls 
Visually 

Blind  [PRE]  _ 

Untimed 
Timed  [PRE] 


Oper  ite  Controls 
Discrete 

Continuous  [PRE]  _ 

Continuous  integrated  [PRE]  _ 

Overall  Difficulty  Factors 

Noise  or  interference  in  environment 

Competing  tasks 

High  stress  on  pilot  (psych) 

High  stress  on  pilot  (phys) 

Timing 

High  Degree  of  Automaticity  required 
Visionless  performance  required 
Length  of  task  procedure  (high  memory  load) 

Highly  branched  task  procedure  (several  alternate  paths) 


Table  7 — Task  element  difficulty  rating  sheet. 
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3.16  Step  16:  Specify  Isolated  Training  Difficulty  Thresholds 

Once  difficult  task  elements  are  identified,  a  determination 
must  be  made  of  those  levels  of  difficulty  which  are  severe 
enough  to  require  isolated  practice  and  mastery  of  the  individual 
sub-elements  before  performance  of  the  entire  task  is  required 
— that  is,  the  threshold  difficulty  level  which  will  cause  the 
developer  to  pretrain  a  subtask  in  isolation  from  its  parent 
task.  Failure  to  identify  and  isolate  such  difficult  task 
elements  results  in  placing  the  student  in  the  situation  of 
learning  both  the  complicated  subtask  and  the  integrated  larger 
task  at  the  same  time. 

Each  element  on  the  Task  Element  Difficulty  Rating  Sheet 
(Table  7)  which  exceeds  the  threshold  set  for  F-16  purposes  is 
marked  with  a  [PRE]  to  indicate  that  isolated  pretraining  of  the 
task  should  be  conducted. 


3.17  Step  17:  Conduct  Extended  Task  Analysis 

Using  the  Task  Element  Difficulty  Rating  Sheet  (Table  7) 
tasks  are  rated  individually  to  identify  their  difficult  elements 
(if  there  are  any).  When  a  [PREl  level  element  is  identified 
within  a  task,  it  is  isolated  as  an  individual,  trainable  task 
itself . 

Analysis  should  be  particularly  aware  of  judging  task 
difficulties  from  the  point  of  view  of  a  new  student  and  should 
reflect  on  their  own  experience  to  find  those  elements  which  made 
it  difficult  when  they  were  learning  complex  tasks.  Levels  of 
difficulty  should  be  judged  for  students  in  early  phases  of 
training  and  not  on  the  basis  of  experienced  pilots.  For  experi¬ 
enced  pilots,  even  difficult  tasks  assume  a  level  of  easiness 
born  of  large  amounts  of  practice. 

Those  elements  identified  and  isolated  for  training  which 
require  equipment  for  training  are  cycled  to  Step  15.  This  means 
that  each  one  will  itself  be  subject  to  analysis  to  determine 
trainable  sub-elements. 

Isolating  subtasks  in  this  manner  constitutes  an  extension 
of  the  task  listing  to  incorporate  tasks  whose  training  impor¬ 
tance  was  nt  readily  apparent  during  the  original  analysis. 


3.18  Step  18:  Insert  New  Tasks  into  Task  Listing 

Each  isolated  element  from  Step  17  in  entered  as  an  inde¬ 
pendent  task  in  the  task  listing.  The  procedure  for  doing  this 
is  found  in  project  report  no.  10,  "Data  Base  Update  Procedures 
Report."  If  this  record  keeping  chore  is  not  completed  the  task 
listing  will  be  incomplete,  and  major  trainable  tasks  will  not 
appear  in  the  syllabus. 
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Steps  15  through  18  may  be  considered  to  be  as  steps  in 
either  the  the  task  analysis  or  sequencing  proc^^ss.  They  are 
included  here  as  part  of  the  media  selection  process  as  a  safety 
precaution  to  back  up  the  task  listing  process,  which  is  often 
subtle  and  can  easily  be  incomplete.  Some  subtasks  which  are  not 
initially  identified  during  task  analysis  may  have  a  high 
training  impact.  Their  omission  may  cause  student  performance 
difficulties  and,  as  a  consequence,  expenditive  of  unnecessary 
personnel  and  training  device  time. 


3.19  Step  19;  Specify  Training  Devices 

When  it  is  determined  that  pre-set  categories  of  training 
devices  will  be  used  as  the  beginning  point  for  device  design, 
the  first  step  is  the  defining  of  the  categories.  Several 
sources  may  suggest  categories:  previous  known  training  df^vice 
configurations;  regulations,  policies,  and  position  papf^rs;  and 
studies  of  training  device  media  characteristics  such  as  the  F-18 
(Hughes,  1977),  B-1  (Sugarman,  1975),  and  TAEG  (Braby,  1965) 
stud  ies . 


Figure  3  is  a  categorization  schema  for  training  df^vices 
based  on  three  character  is  i  t  ics  of  devices:  (1)  d  imf'ns  ional  i  ty 
of  the  display  presented  to  the  stud#^nt  (2  vs.  3),  (2) 
responsiveness  to  student  manipulation  (responsive  vs.  non-),  and 
(3)  scope  of  simulation  of  system  (isolated  systems  vs. 
integrated  and  full  systems).  The  major  categories  of  devices 
resulting  from  crossing  these  characteristics  are  df^scribed 
below.  This  category  scheme  will  be  used  for  F-16  purposes. 

A .  Two-dimensional,  non-interactive  display  in  n>^ar-rpal 
size,  proportion,  and  degree  of  realistic  appearance. 

An  example  of  this  device  would  be  a  photographed  or 
line-drawn  display  of  a  cockpit  interior  pasted  upon  a 
wood  or  cardboard  surface  in  such  a  way  that  preserved 
spatial  and  proportional  relationships  of  individual 
components  and  panels  (paper  tiger). 

B .  Two-dimensional,  responsive  schematic  panel  consisting 
of  one  isolated  system.  This  device  makes  available  to 
the  student  isolated  system  to  be  manipulated.  Through 
a  two-dimensional  display  manipulations  of  the  system 
are  not  topologically  the  same  as  real-world  behavior 
but  can  be  cognitively  equivalent.  An  example  of  this 
type  device  would  be  a  two-dimensional  CRT  presentation 
of  a  given  aircraft  control  panel  driven  by  a  computer 
and  capable  of  simulating  control  panel  action  and 
reaction  with  high  fidelity. 

C .  Two-dimensional,  interactive,  partially  to  fully 
Integrated  schematic  panels.  In  this  device  a  number  of 
systems  are  integrated  in  one  instructional  training 
device  which  operates  the  same  as  device  "B" .  This 
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device,  however,  is  capable  of  simulating  the  operation 
of  multiple  systems  and  their  interactions  with  high 
f idel ity . 


D .  Three-dimensional,  non-interact ive  model  of  isolated 
systems .  Parts  of  this  model  may  be  movable  manually 
]ust  as  real  equipment,  but  there  is  no  automated 
modeling  of  responses  to  user  manipulations.  One 
example  would  be  a  model  of  a  physical  system  or  a  non¬ 
physical  phenomenon  intended  to  show  operation  of  parts 
during  a  cycle  of  activity.  A  second  example  would 
include  isolated  pieces  of  equipment  intended  for  mani¬ 
pulation,  such  as  realistic  models  of  equipment  to  be 
repaired  or  cockpit  panels  and  equipment  taken  from  or 
resembling  an  aircraft  system. 

E .  Three-dimensional,  non-interactive  model  of  multiple 
systems .  Parts  of  the  model  may  be  made  to  move  in  a 
way  simulating  real  motions  within  the  object  during 
operation,  but  modeling  of  responses  to  user  manipula¬ 
tions  is  not  made.  An  example  of  this  system  is  the 
dead  A/C  system  or  a  CFT,  representing  physical  aspects 
of  multiple  systems. 

F .  Three-dimensional,  interactive,  limited  scope.  This  is 
a  simulation  of  a  realistic  (within  some  range  of  fidel¬ 
ity  to  real  world  operation)  performance  environment  in 
which  only  selected  portions  of  the  environment  are 
simulated.  Examples  of  this  include  isolated  equipment 
systems  placed  in  working  order  or  combinations  of 
systems  in  working  order  either  separately  or  with 
interactions  betw<jen  them  being  duplicated.  Examples 
also  include  the  Egress  Procedures  Trainer. 

G .  Three-dimensional,  interactive,  limited  to  broad  scope 
simulation  of  a  realistic  performance  environment  in 
which  either  extensive  or  all  portions  of  the  environ¬ 
ment  are  simulated.  Examples  of  this  type  range  from 
limited  operational  environmental  scope  to  operational 
flight  trainers  which  simulate  to  some  degree  of  fidel¬ 
ity  a  real  operational  environment.  The  IPTT,  OFT  and 
WST  are  examples. 

H.  F-16  Aircraft. 


3.20  Step  20;  Assign  Tasks  to  Training  Devices 

During  this  step  a  preliminary  decision  will  be  made  on  two 
points:  (a)  on  which  training  devices  is  it  possible  to  execute 

each  task  at  some  level  and  (b)  the  estimated  desirability  of 
having  students  perfom  each  task  in  a  given  training  device  dur¬ 
ing  actual  training.  This  step  approximates  closely  an  activity 
which  will  also  go  on  during  syllabus  building:  That  of  assign- 
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Figure  3  —  Device  Type  Classifications 


ing  tasks  to  specific  training  devices  for  practice  purposes. 

The  products  of  this  estimation  for  media  selection  purposes  will 
become  the  inputs  to  syllabus  building. 


During  this  step  it  must  be  indicated,  for  each  task, 
whether  or  not  the  task  can  be  performed  to  some  level  in  a  given 
training  device.  This  indication  is  made  in  the  columns  labelled 
"Training  Device  Capability  and  Desirabilities"  on  the  Task 
Listing  Data  Worksheet  (Table  8)  by  placing  a  check  in  the  column 
for  each  device  on  which  the  task  can  performed, 

A  built-in  assumption  for  each  training  device  type  is  that 
only  certain  varieties  and  levels  of  practice  will  be  attainable. 

A  cockpit  procedures  trainer,  for  instance,  will  not  allow  a  j 

student  timed  performance  of  a  basic  procedure  under  full  stim¬ 
ulus  conditions.  However,  some  preliminary  practice  is  possible 
on  the  mechanics  of  the  procedure.  As  decisions  are  made 
concerning  which  device  may  be  used  for  each  task  during  this 
step,  it  should  be  kept  in  mind  for  each  device  what  level  of 
practice  is  possible.  It  may  be  desirable  to  record  for  each 
device  type  the  kind  of  practice  possible. 

The  second  activity  in  this  step,  using  the  same  form,  is  to  i 

specify,  given  the  entry-level  behavior  of  the  student  population 
and  the  course  mastery  model,  the  likelihood  that  it  will  be  f' 

desirable  from  the  student  point  of  view  to  actually  use  the 

training  devices  indicated  in  the  first  part  of  this  step.  Some  ' 

guidelines  for  making  this  decision  are:  (1)  tasks  should  be 

practiced  in  the  least  complex  simulator  (to  lower  overall 

training  costs)  ,  (2)  practice  in  simple  environments  on  tasks  for 

which  students  have  had  some  degree  of  previous  experience  should 

be  minimized  or  omitted  to  avoid  wasteful  and  unnecessary 

training  time,  and  (3)  practice  should  always  be  escalated  up  to 

the  level  of  training  device  which  can  most  fully  simulate  actual 

conditions  which  will  occur  on  the  job  during  performance  of  the 

task . 


Indication  of  the  desired  training  devices  for  a  given  task 
is  made  by  placing  a  circling  around  the  x's  made  earlier  on 
Table  8  in  columns  which  indicate  training  devices  should 
actually  be  used  for  a  task.  These  circles  indicate  the  task 
assignments  which  will  be  used  to  determine  number  of  training 
devices  required  and  cost  trade-offs  to  be  used  in  future  steps. 

It  is  necessary  to  specify  the  systems  required  for  perform¬ 
ance  of  each  task.  This  information  is  available  on  the  criter¬ 
ion  referenced  objectives  (CRO's).  A  check  should  be  conducted 
to  insure  that  each  CRO  contains  this  information.  Then  indica¬ 
tion  of  the  systems  required  for  the  task  is  made  by  checking  the 
appropriate  columns  under  the  heading  "Systems  Required"  on  Table 
8.  Under  this  general  heading,  each  control  and  indicator  or 
panel  accessible  to  the  pilot  should  be  placed  at  the  top  of  a 
column.  To  economize  effort,  it  is  recommended  that  panels  be 
used  as  the  unit  of  indication  where  a  control  or  indicator 
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TRAINING  DEVICE  CAPABILITY  SYSTEMS  REQUlflED 

AND  desirability 


Table  8  Ta%k  tistmg  Data  Worksheet 


Tables  (Con't) 


appears  as  part  of  a  panel.  For  controls  and  indicators  which 
occur  isolated  and  not  part  of  a  panel,  it  is  recommended  these 
be  listed  separately. 


3.21  Step  21;  Specify  Possible  Configurations  for  Each  Training 
Device  Type 

During  this  step  a  list  is  made  of  training  devices  required 
to  carry  out  the  practice  needs  indicated  in  Step  20.  This  is 
not  the  final  list  of  devices  which  will  be  obtained  but  rather  a 
preliminary  working  list.  This  listing  is  made  by  identifying 
all  device  types  and  the  versions  of  each  which  are  indicated  by 
the  marks  on  Table  8.  A  recommended  procedure  is: 

1.  Place  the  name  of  each  device  type  at  the  head  of  a 
separate  training  device  configuration  record  (Table  9). 

2.  For  device  types  which  span  only  one  or  a  small  number 
of  systems  (the  "isolated  systems"  devices  from  Step 
20;  this  would  be  devices  B,  D,  and  E)  an  entry  is  made 
on  the  appropriate  record  to  show  which  isolated  system 
must  be  simulated.  Each  unique  system  or  set  of  systems 
is  given  a  configuration  number  and  treated  as  a 
separate  device.  The  frequency  of  the  requirement 
should  be  shown  for  each  item. 

3.  The  device  types  which  simulate  a  larger  range  of 
systems,  those  combinations  of  systems  called  for  on 
Table  8  should  be  listed,  along  with  the  frequency  with 
which  the  combination  is  called  for  on  Table  8.  Each 
unique  set  of  systems  is  given  a  configuration  number 
and  treated  as  a  separate  device. 

A  Training  Device  Configuration  Worksheet  like  that  in  Table 
10  is  filled  out  for  each  entry  in  Table  9.  This  worksheet  will 
serve  as  a  gathering  point  for  data  on  specific  device  configura¬ 
tions  as  they  evolve.  All  available  data  is  recorded  on  the 
worksheet.  At  this  point,  systems  required  should  be  noted  in 
the  appropriate  space. 


3.22  Step  22;  Determine  Instructional  Strategies  to  be  Used  in 
Each  Training  Device  and  the  Potential  Instruc¬ 
tional  Features  for  Each  Training  Device 

The  same  training  device  may  be  used  in  several  different 
ways  instr uct ional ly .  Some  plans  for  device  use  entail  a  high 
degree  of  instructor  involvement  and  interaction  with  the 
student.  Others  require  the  student  to  work  alone.  Still  others 
require  presentation  of  mediated  instruction  by  the  device. 
Moreover,  it  may  be  desirable  for  the  device  to  demonstrate  a 
procedure  to  the  student,  recall  a  student  performance  and 
display  it,  or  stop  execution  of  a  task  and  reset  to  beginning 
cond i t ions . 
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TRAINING  DEVICE  CONFIGURATION  RECORD 


Table  9 — Training  device  configuration  record. 


Performance 

Measurement 

Features 


Instructional 

Features 


REQUIRED  SYSTEMS 


Table  10  -  Training  Device  Configuration  Worksheet 


Since  devices  are  normally  used  in  the  context  of  a  training 
session,  the  pattern  of  a  device  session  should  be  planned  at 
this  time  at  a  general  level  if  that  did  not  already  occur  during 
Step  20.  The  profile  of  this  generic  device  session  will  recom¬ 
mend  particular  instructional  characteristics  for  each  device 
which  should  be  listed  at  this  time.  Desired  features  may 
include  some  of  the  following:  special  provision  for  student- 
instructor  communication,  special  instructor  displays  showing 
student  control  displays  or  student  performance  summaries, 
special  positioning  of  the  instructor,  instructional  feature 
control  access  for  students  and  instructor  (for  control  of  pace, 
exercise  termination,  etc.),  special  provision  for  problem  set 
up,  special  provision  for  briefing  and  debriefing  capabilities, 
or  provided  instructor  working  space. 

Table  11  may  be  used  as  a  worksheet  to  generate  this  infor¬ 
mation.  Session  type  and  session  activities  sequence  is  recorded 
for  each  device  type.  Space  is  provided  to  record  the  instruc¬ 
tional  features  which  correspond  to  each  part  of  the  sequence. 
Once  desired  instructional  features  have  been  selected,  they 
should  be  entered  onto  the  appropriate  Training  Device  Configura¬ 
tion  Worksheet  generated  in  Step  21  (Table  10) . 


3.23  Step  23;  Determine  Performance  Measurement  to  be  Executed 
in  Training  Devices  and  the  Potential  Performance 
Measurement  Features  for  Each  Training  Device 

Performance  measurement  may  be  instructor-administered  or 
machine-administered.  It  may  measure  tolerances,  specific 
values,  steps  in  a  procedure,  or  logical  conditions.  For  each 
training  task  a  standard  will  have  been  written  prior  to  this 
step  which  states  the  performance  level  expected  of  the  student. 
This  standard  will  suggest  what  must  be  measured  during  perform¬ 
ance  measurement.  During  this  step  for  each  task  a  decision  is 
made  as  to  the  evaluation  agent  (instructor/student/training 
device)  and  the  variables  to  be  monitored  during  evaluation 
(tolerance  bands,  times  in  tolerance,  trends,  paths,  etc.).  This 
information  is  recorded  on  the  appropriate  Training  Device 
Configuration  Worksheet  from  Step  20  (Table  10). 

At  the  same  time  as  measurement  agents  and  variables  are 
specified,  the  form  of  the  desired  measurement  output  must  be 
specified  to  the  extent  it  is  known.  That  output  may  consist  of 
printouts  or  readouts  of  various  kinds  and  at  various  levels  of 
sophistication.  Preferences  on  output  are  also  entered  in  the 
device  worksheets. 


3.24  Step  24:  Estimate  the  Cost  of  Each  Configuration  for  Each 
Training  Device 

There  are  several  sources  that  may  be  contacted  for  making 
cost  estimations  of  the  various  device  configurations.  They  are 
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TABLE  FOR  RECORDING  INSTRUCTIONAL  STRATEGIES 
AND  FEATURES  FOR  EACH  TRAINING  DEVICE 


Table  ll--Table  for  recording  instructional 
strategies  and  features  for  each  training  device 


(a)  other  training  systems  which  have  recently  acquired  similar 
equipment,  ( b)  training  device  contractors,  or  (c)  training 
device  procurement  agencies  within  the  military.  These  estimates 
are  registered  for  each  device  on  the  appropriate  Training  Device 
Configuration  Worksheet  (Table  10).  Estimates  should  take  into 
consideration  the  cost  of  desired  instructional  and  performance 
measurement  features  for  each  training  device  and  represent  cost 
breakdowns  where  possible. 


3.25  Step  25:  Determining  Time  Requirements  for  Each  Configura¬ 
tion  of  Each  Training  Device 

The  total  amount  of  time  required  for  each  training  device 
must  be  determined.  This  time  will  be  found  by  adding  up  the 
time  requirements  for  each  task  assigned  to  one  device.  The 
following  data  must  be  collected  for  each  task  and  entered  in  the 
appropriate  columns  in  Table  8: 

Time  to  pe  r  fo  rm- - thi  s  figure  represents  the  amount  of  time 
that  it  takes  to  perform  the  task  once  through.  Since 
training  devices  are  used  for  new  students  and  proficient 
students  and  pilots  both,  the  time  is  determined  by  taking 
the  amount  of  time  estimated  for  the  student's  first 
performance  of  the  task,  adding  that  to  the  time  estimated 
for  a  competent  performer  to  execute  the  task,  and  then 
dividing  by  two.  This  gives  a  rough  initial  estimate  of  the 
average  amount  of  time  required  to  perform  the  task  using 
the  equi  pment  . 

Time  to  set  up--this  figure  represents  the  amount  of  time 
required  to  set  up  the  training  device  prior  to  each  task 
exec  ut  ion . 

Number  of  grouped  execut  ions--When  a  task  is  performed 
during  training,  it  most  likely  will  not  be  performed  once 
only.  This  figure  is  an  estimate  of  the  number  of  times 
that  the  task  will  be  repeated  when  it  is  being  practiced, 
averaged  over  the  training  span  for  that  task.  The  training 
span  begins  with  first  practice  of  the  task  and  ends  with 
evaluation  and  certification  of  a  task  in  a  given  training 
dev  ice  . 

Number  of  practice  sessions--rh i s  figure  estimates  the 
number  of  times  massed  practices  should  be  repeated — that 
is,  the  number  of  different  training  sessions  in  which  the 
student  will  perform  the  task  prior  to  the  end  of  the 
training  span. 

Number  of  evaluations — Evaluation  and  certification  of  task 
performance  adequacy  in  a  given  training  device  ends  the 
training  span  for  the  task  in  a  given  device.  This  figure 
gives  the  number  of  evaluation  trails  required  before  a 
given  task  may  be  assumed  mastered.  The  figure  will 
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increase  for  more  difficult,  more  complex,  or  more  critical 
tasks  . 

Number  of  review  practices  student  per  year--Tnis  figure 
states  the  number  of  review  practices  required  by  each 
student  and  course  graduate  using  a  given  training  device 
per  year . 

All  of  the  figures  called  for  are  difficult  to  produce. 
Estimates  may  be  based  upon  subject  matter  exterts'  best 
knowledge  and  experience,  and  should  represent  the  amount  of 
practice  required  to  bring  a  student  to  the  desired  standard  of 
behavior  (as  stated  in  the  course  mastery  model,  see  project 
report  no.  23,  “F-i6  Instructional  Sequencing  Plan  Report.") 

More  sophisticated  ways  of  determining  time  to  perform  can 
be  implemented  once  training  equipment  is  in  use.  Data  may  be 
gathered  from  pilots  actually  executing  a  timed  performance  of 
each  task  and  then  fed  back  to  the  media  selection  process. 

Steps  30  and  31  in  this  media  selection  sequence  are  designed  to 
provide  a  feedback  loop  to  the  media  selection  process.  As  real 
data  begins  to  be  gathered  on  actual  times  to  train,  estimates 
made  initially  for  media  selection  can  give  way  to  more  accurate 
figures,  benefitting  not  only  F-16  media  selections  but  future 
aircrew  training  selections  as  well.  For  obtaining  the  initial 
figures,  however,  the  estimating  method  appears  to  be  the  most 
practical  one. 

The  total  time  required  for  each  trai.iing  device  is  found  by 
adding  the  total  training  times  for  task<;  assigned  to  each  and 
multiplying  by  the  yearly  flow  of  students.  Task  training  times 
are  found  by  multiplying  and  adding  times  to  perform  and  repeti¬ 
tion  together  appropriately.  The  result  of  this  step  will  be  a 
total  amount  of  time  required  for  use  for  each  training  device 
1 isted  in  Step  20. 


3.26  Step  26:  Consolidate  Configurations  Using  Time  Factors 

A  consolidation  is  made  at  this  time  to  eliminate  unneces¬ 
sary  devices  and  economically  unjustifiable  ones  by  combining 
them  where  possible. 

The  total  time  demanded  for  one  training  device  is  deter¬ 
mined  by  adding  the  total  times  for  tasks  assigned  to  the  device. 
Since  the  time  obtained  in  this  addition  represents  the  training 
time  on  the  device  required  for  one  student,  each  total  must  be 
multiplied  by  the  anticipated  annual  student  throughput.  The 
resultant  figure  indicates  the  number  of  minutes  per  year  the 
device  will  be  required  to  supply  strictly  for  training. 

To  determine  whether  or  not  that  amount  of  time  is 
sufficient  to  warrant  a  training  device  follow  this  procedure: 
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1.  Take  the  total  operating  day  time  for  the  device  and 
subtact  maintenance  time,  down  time,  and  time  expected 
to  be  allocated  for  other  training  courses  and  multiply 
this  by  the  number  of  training  days  per  year  to  obtain 
the  total  available  time  for  training  with  the  device. 

2.  Divide  the  total  required  time  for  the  device  by  the 
total  available  time,  calculated  in  no.  1.  This  number 
indicates  the  number  of  training  devices  required  for 
each  configuration. 

If  the  above  calculated  number  is  within  a  given  range  (.85 
and  1.00  for  F-16)  the  configuration  should  stand  alone.  If  the 
calculated  number  is  less  than  the  lower  range  value  tasks  from 
the  configuration  can  be  combined  with  another  configuration  in 
which  the  required  systems  are  available  or  can  be  made  available 
most  easily.  This  consolidation  of  training  devices  should  con¬ 
tinue  until  all  devices  remaining  are  rated  within  the  desirable 
range.  Ca  e  should  be  taken  to  see  that  as  devices  are  consoli¬ 
dated  that  important  performance  measurement  and  instructional 
features  are  also  consolidated,  and  time  required  for  consoli¬ 
dated  devices  should  be  calculated  and  subjected  to  a  test  of 
ratios  to  insure  that  they  fall  within  the  justification  range. 
Also,  as  consolidations  are  made,  the  costs  of  the  consolidated 
device  must  be  estimated. 

The  effects  of  the  setting  of  the  range  should  be  noted. 

For  a  high,  immediate  student  surge  capability  in  the  instruc¬ 
tional  system  the  range  should  be  kept  low  for  more  complex,  more 
difficult,  and  expensive  to  obtain  training  devices.  If  training 
devices  are  fully  scheduled  or  over-scheduled  there  is  little 
room  for  expansion  of  the  student  number  without  decrementing 
training.  A  balance  must  be  set  between  having  some  training 
device  time  available  in  reserve  for  use  for  a  surge  and  allowing 
training  devices  to  go  unused. 


3.27  Step  27:  Determine  Cost  for  Total  Training  Device  Plan 

The  necessary  information  to  complete  this  step  has  been 
determined  in  previous  steps.  The  cost  per  device  configuration 
was  estimated  in  Step  24  and  entered  on  the  Training  Device 
Configuration  Worksheet  (Figure  9),  The  number  of  confi  ug  rations 
needed  was  determined  in  Step  26,  and  revised  cost  figures  were 
g  ener ated . 

To  determine  the  cost  for  the  total  training  device  plan, 
the  costs  of  the  training  devices  left  in  consideration  should  be 
multiplied  by  the  number  of  devices  required  and  the  total  cost 
for  all  devices  added  up. 
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3.28  Step  28: 


Adjust  Assignments  for  Best  Total  Training  Device 
Plan  Cost 


Adjustments  should  be  made  at  this  point  to  increase  the 
practicality  of  the  training  device  plan.  Since  cost  is  usually 
the  main  constraining  factor  in  training  device  designs,  it  may 
be  used  as  a  filter  of  the  practicality  of  the  plan.  Adjustments 
to  the  total  plan,  however,  should  be  accompanied  by  correspond¬ 
ing  shifts  in  training  events  to  acceptable  training  alternate 
media  or  corresponding  reductions  in  expected  performance  from 
course  graduates.  It  should  not  be  assumed  that  simply  reducing 
training  devices  is  making  the  training  more  efficient,  since 
efficiency  cannot  be  guaranteed  unless  effectiveness  is  first 
obtained . 


3.29  Step  29:  Write  Training  Device  Specifications 

Once  a  satisfactory  assignment  of  tasks  to  training  device 
configurations  has  been  reached  and  a  desirable  group  of  devices 
is  selected,  it  is  necessary  to  prepare  a  more  det^lxed  specifi¬ 
cation  of  the  required  capabilities  of  each  training  device.  The 
instructional  presentation  capabilities  and  the  hardware/software 
capabilities  should  be  specified  in  detail.  These  descriptions 
should  be  complete  so  that  they  can  serve  as  input  to  the 
engineering  personnel  responsible  for  development  of  the  trainer 
functional  specifications. 
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3.30  Step  30:  Collect  Time  to  Proficiency  and  Level  of  Use  Data 
on  Training  Devices 

Once  the  training  devices  have  been  implemented,  it  is 
necessary  to  collect  information  pertinent  to  the  accuracy  of 
original  estimates. 

1.  The  original  time  estimates  must  be  validated.  This 
task  can  be  carried  out  as  part  of  the  evaluation 
activity  of  the  instructional  system.  The  actual  time 
to  perform  tasks  can  be  averaged  across  a  class  of 
students  to  use  in  comparison  with  the  original 
estimates . 


2. 


Data  should  be  collected  on  the 
training  device. 
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3.31  Step  31:  Adjust  Training  Device  Selection  Data  Base 


The  data  gathered  in  the  previous  step  is  fed  back  into  the 
media  selection  process.  This  information  is  used  to  correct  the 
present  media  selection  process  as  much  as  possible  and  guide  any 
needed  task  reallocations  between  devices.  It  is  also  added  to  a 
data  pool  for  future  media  selection  device  type  and  amount 
levels.  Over  time,  as  the  data  base  builds,  it  can  provide 
instructional  developers  of  other  weapon  systems  a  valuable  pool 
of  information  out  of  which  syllabus  parameters  may  be  derived. 
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ATTACHMENT  I 


F-16  ACADEMIC  OBJECTIVE  CATEGORIES  AND 
INSTRUCTIONAL  STRATEGIES 


I-l.O  INTRODUCTION 


This  section  enumerates  the  academic  objective  types  used 
during  objectives  hierarchy  analysis  for  the  F-16  project.  There 
were  nine  classes  of  objectives  used: 

Remember /Procedure  (verbatim) 

Remember/Procedure  (paraphrase) 

Remember/Fact  (symbol/name) 

Remember /Fact  (name/f unct ion/ location) 

Remember /Fact  (limits/tolerances/critical  values) 
Remember /Fact  (organized  lists) 

Remember /Fact  (sequences) 

Use/Concept 

Use/Rule 

To  place  these  classes  in  perspective,  they  are  related  in  Figure 
I-l  to  a  simplified  category  system  for  academic  objectives.  In 
this  system  there  are  two  levels  of  objective:  (1)  those  which 
require  exercise  of  some  form  of  memory  recall,  labelled  "remem¬ 
ber"  objectives  and  (2)  those  which  require  application  of  some 
variety  of  intellectual  or  procedural  skill,  labelled  "use" 
objectives.  These  two  levels  are  crossed  with  four  varieties  of 
instructional  content:  fact,  concept,  rule,  and  procedure.  This 
cross  produces  the  eight  general  types  of  academic  instructional 
objectives  to  which  the  F-16  objectives  types  relate  directly  as 
shown . 

For  the  development  of  F-16  instruction,  each  objective  type 
corresponds  to  an  instructional  strategy.  For  each  strategy  the 
following  are  specified:  (1)  a  set  of  instructional  display 
types  to  be  made  available  to  the  student  in  instruction  and  (2) 
a  prescription  for  the  manner  in  which  these  displays  should  be 
presented  to  the  student,  including  prescriptions  for  learner 
versus  system  control,  pace  and  timing,  and  other  options  avail¬ 
able  to  either  the  student  or  the  instructional  device.  The 
instructional  strategy  for  each  of  the  objective  types  is 
described  below. 


I-l.l  REMEMBER/PROCEDURE  (VERBATIM)  OBJECTIVES 


I-l. 1.1  Objective  Form 

Remember/Procedure  (verbatim)  objectives  are  those  which  are 
worded:  "State  the  procedure  for  _ Objectives  of 
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\'ARirTIl-S  or  INSTRIICTTOXAI.  CONTIi.NT 


-ori^anircd  lists 
-  sccjucnccs 


this  type  are  written  for  those  procedures  which  the  student  must 
know  verbatim  and  from  memory. 


I-1.1.2  F-16  Example 

Examples  of  F-16  objectives  which  will  fall  into  this 
category  are  "know  cold"  emergency  procedures:  "know  cold" 
checklists;  and  complex,  high  memory  load  procedures  which  must 
be  committed  to  memory  before  performance  is  attempted. 


I-1.1.3  Strategy  Components 

Components  which  will  be  used  for  constructing  strategies 
for  this  type  of  objective  will  be: 

Generality — the  information  to  be  memorized. 

Mnemonic  (generality  help)--a  specific  verbal  or  pictorial 
device  designed  as  an  aid  to  memory. 

Demonstration — a  recorded  or  live  execution  of  the  procedure 
step-by-step,  complete  with  initiation  cues,  processes, 
and  termination  cues  defined  for  each  step. 

Drill  and  practice--an  exercise  requiring  students  to  respond 
to  a  particular  memory  stimulus  before  a  timeout  proce¬ 
dure  disqualifies  answering.  Such  procedures  can  be 
conducted  by  the  student  in  a  self-monitoring  mode,  by 
instructors,  by  other  students,  or  by  a  device  which  has 
a  timeout  capability. 


I-1.1.4  Control  and  Pace  Considerations 

Control  and  pace  considerations  required  for  the  administra¬ 
tion  of  this  strategy  are  centered  in  the  requirement  for  a  timeout 
procedure  to  be  exercised  as  the  student  uses  the  drill  and  prac¬ 
tice.  In  considering  which  of  the  various  options  should  be  used 
to  administer  the  drill  and  practice  (self,  instructor,  other 
students)  allowance  should  be  made  for  those  options  which  are  less 
expensive  (self,  other  student)  and  those  which  are  most  likely  to 
recieve  high  student  acceptance  (self,  instructor,  media  device). 


I-1.1.5  Display  Considerations 

For  the  generality  and  the  mnemonic  there  must  be  the  capabil¬ 
ity  to  present  verbal  and  pictorial  information  of  an  uncomplicated 
type  if  the  material  to  be  memorized  is  primarily  verbal  in  nature. 
The  realism  and  technical  detail  of  these  displays  is  important 
only  in  cases  where  the  generality  contains  a  complex  graphic 
representation  to  be  committed  to  memory.  During  drill  and  prac¬ 
tice  the  same  display  considerations  hold. 


1-1. 1.6  Response,  Scoring,  and  Feedback  Considerations 

The  response  considerations  for  this  strategy  require  that 
some  monitoring  or  recording  capability  must  be  available  to  the 
student.  It  must  be  capable  of  accepting  binary  (yes/no)  or  verbal 
responses,  grading  them,  and  providing  feedback  on  their 
correctness  within  a  matter  of  seconds. 


1-1.2  REMEMBER/PROCEDURE  (PARAPHRASE)  OBJECTIVES 


I  -1.2.1  Objective  Form 

Remember/Procedure  (paraphrase)  objectives  will  be  worded: 

"State  the  procedure  for _ ",  or  "Given  a  list  of  steps,  some 

of  which  are  steps  in  the  _ procedure,  select  those  steps 

which  are  in  the  _ procedure  and  arrange  them  in  the  proper 

order",  or  "Given  a  statement  of  the  steps  in  the  _ 

procedure,  determine  whether  all  steps  are  present  and  all 
expressed  critical  values  accurate."  This  type  of  objective 
differs  from  Remember  Procedure  (Verbatim)  in  that  some  "under¬ 
standing"  on  the  part  of  the  student  is  expected  in  paraphrase 
recall,  whereas  rote  memorization  is  all  that  is  required  for 
verbatim  recall.  Although  the  student  may  "understand"  rotely 
memorized  procedures,  that  is  of  secondary  importance  to  the 
student's  ability  to  recall  the  procedure  without  hesitation  and 
exactly . 


1-1.2. 2  F-16  Example 

Examples  of  F-16  objectives  which  fall  into  this  category 
include  most  of  the  procedures  which  will  be  taught,  including 
takeoffs  and  landings,  refueling  procedures,  approach  and  navi¬ 
gation  procedures,  and  most  checklists  and  emergency  procedures, 
specifically  those  which  are  not  "know  cold".  These  objectives 
constitute  a  large  portion  of  the  F-16  objectives. 


1-1.2. 3  Strategy  Components 

The  components  of  strategy  which  will  be  used  for  this  type  of 
objective  are: 

Generality — a  verbal  statement  of  the  steps  in  the  procedure. 

Demonstration — a  recorded  or  live  execution  of  the  procedure 
step-by-step,  complete  with  initiation  cues,  processes, 
and  termination  cues  defined  for  each  step. 

Cause/effect  (Generality  Help)--a  display  showing  within- 
system  responses  to  manipulations  during  a  procedure. 

This  is  to  be  used  where  appropriate  as  a  mnemonic  for 
procedure  recall. 
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Practice  help — assistance  to  the  student  in  determining 
misconceptions  or  mis- info rmation  causing  incorrect 
practice 

1-1.2. 4  Control  and  Pace  Considerations 

Control  for  this  strategy  may  be  entirely  in  charge  of  the 
student  since  there  is  no  requirement  for  pacing  or  for  a  timed- 
response  input. 


1-1.2. 5  Display  Considerations 

Display  considerations  for  the  generality  require  the  presen¬ 
tation  of  verbal  information  either  auditorily  or  in  print.  Demon¬ 
strations  for  some  procedures  will  require  no  more  than  a  step-by- 
step  graphic  illustration,  particularly  where  the  procedure  is  made 
up  of  relatively  common  or  already  well-mastered  student  behaviors. 
In  some  cases  the  demonstration  will  require  highly  detailed  still- 
illustrated  or  motion- il 1 ustrated  presentations  of  procedures  being 
performed.  Live  demonstrations  may  be  appropriate  here,  but  care 
should  be  taken  of  the  difference  in  information  processing  loads 
placed  upon  the  student  by  live  demonstrations  not  placed  upon  him 
by  recorded  demonstrations.  Note  should  also  be  made  of  the  fact 
that  most  demonstrations  performed  live  are  difficult  or  expensive 
to  playback  or  repeat  due  to  operational  factors.  Procedures 
especially  requiring  more  extensive  demonstrations  will  be  those 
involving  behaviors  new  to  the  student,  critically-timed,  or 
requiring  difficult  or  tricky  manipulations. 


I-1.2.6  Response,  Scoring,  and  Feedback  Considerations 

Response  considerations  for  these  objectives  are  less  sophis¬ 
ticated  than  for  verbatim  recall  objectives.  Students  have  no  need 
to  make  timed  responses  or  to  re::eive  timed  feedback  from  the 
instructional  device.  An  answer  key  used  by  the  student  to  correct 
his  own  responses  is  sufficient  as  a  minimum.  An  instructor  may  be 
used  for  this  function,  entering  into  a  dialogue  with  the  student. 
Feedback  can  be  provided  as  a  part  of  an  answer  key  or  by  an 
instructor . 


1-1.3  REMEMBER/FACT  (SYMBOL/NAME) 


I-1.3.1  Objective  Form 

Remember/Fact  (symbol/name)  objectives  are  those  which  are 

worded;  "Given  to  symbol  (or  signal)  for  _ ,  produce  the 

name  or  meaning  of  the  symbol,"  or  "Given  ^Ke  name  or  meaning  of 
a  symbol  (or  signal),  produce  or  identify  the  symbol." 
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1-1.3. 2  F-16  Example 


Examples  of  F-16  objectives  of  this  type  are  those  dealing 
with  hand  signals  both  from  ground  crew  members  and  for  use 
between  members  of  a  flight.  Objectives  on  visual  aircraft, 
ship,  and  ground  installation  or  map  symbol  identification  fall 
into  this  category.  These  objectives  include  also  symbology  used 
on  HUD  or  radar  displays  and  attached  meanings. 


1-1.3. 3  Strategy  Components 

The  components  of  stategy  which  will  be  used  with  this  type 
of  objective  include: 

Generality — a  verbal  and  graphic  statement  of  each  symbol  and 
its  associated  name  or  meaning. 

Mnemonic  (Generality  Help) --a  verbal  or  graphic  memory  aid 
where  appropriate. 

Drill  and  practice--a  timed-response  drill  of  the  type 
described  in  Section  I-1.1.3. 


1-1.3. 4  Control  and  Pace  Considerations 

The  drill  and  practice  required  for  this  type  of  instruction 
is  timed  practice  like  that  described  in  Section  I-1.1.4  for 
Remember/Procedure  (verbatim)  strategies.  The  need  for  periodic 
review  and  practice  which  is  common  to  verbatim  memorizing  must  be 
taken  into  account  when  assessing  the  amount  of  instruction  of  this 
variety. 


1-1.3. 5  Display  Considerations 

Display  considerations  for  this  type  of  objective  must  take 
into  account  the  fact  that  body  motions  are  in  some  cases  the 
subject  matter.  For  some  hand  signals,  for  instance  motion  is 
required  for  displaying  the  signals,  particularly  those  that  are 
unusual  and  where  the  signal  cannot  be  represented  unambiguously  in 
non-motion  media. 


1-1.3. 6  Response,  Scoring,  and  Feedback  Considerations 

Response  considerations  for  instructional  strategies  involving 
drill  and  practice  where  the  stimulus  involves  motion  are  much  more 
demanding  if  mediated  instructional  means  are  used.  The  use  of 
instructors  seems  to  be  a  reasonable  way  to  administer  instruction 
such  as  this,  given  the  provision  that  instruction  must  take  place 
in  group  mode  for  it  to  be  efficient,  and  students  must  take 
responsibility  for  their  own  drill  and  practice.  Some  media 
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devices  are  capable  of  administering  this  type  of  instruction  and 
monitoring  drill  and  practice  also. 

In  drill  and  practice  routines  which  are  machine  controlled, 
scoring  considerations  become  important  and  some  mechanism  must  be 
included  in  the  medium  for  scoring  student  responses  in  a  cumula¬ 
tive  fashion.  This  is  especially  true  for  each  item  being  memor¬ 
ized  if  a  replacement- type  strategy  is  used  in  which  elements  which 
have  been  memorized  are  dropped  from  the  list  and  new  items  are 
added . 


1-1.4  REMEMBER/FACT  (NAME/FUNCTION/LOCATION) 


I-1.4.1  Objective  Form 

Remember/Fact  (name/function/location)  objectives  are  those 
which  are  worded:  "Given  the  name  of  an  object  (or  system),  state 
its  function  and  location",  or  "Given  the  function  of  an  object  (or 
system)  produce  its  name  and  location".  That  is,  given  any  one  of 
the  three  elements  of  information  this  student  is  required  to 
supply  the  other  two  in  some  mode  of  responding. 


1-1.4. 2  F-16  Example 

Examples  of  this  type  of  instruction  relative  to  the  F-16  are 
found  in  objectives  which  pertain  to  the  location  of  controls 
within  the  cockpit,  inspection  points  on  the  aircraft,  and  the 
location  and  function  aircraft  subsystems  such  as  the  radar  or  HUD. 


1-1.4. 3  Strategy  Components 

The  strategy  components  for  this  type  of  objective  will 
include : 

Generality--a  verbal  and  graphic  statement  of  the  name, 
function  and  location  data  to  be  learned. 

Practice — a  set  of  verbal  exercises  accompanied  by  a  set  of 
graphic  memory  exercises.  A  variety  of  devices  may  be 
used  to  create  these  exercises  such  as  those  described 
in  Section  1-1.2. 3. 

Practice  help — same  as  "practice  help"  under  Section 

1-1.2. 3. 


1-1. 4. 4  Control  and  Considerations 

Control  and  pace  considerations  for  this  type  of  instruction 
are  the  same  as  described  in  Section  1-1.2. 4,  concerning  Remember/ 
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Procedure  (paraphased)  objectives.  Time  drills  may  be  used  which 
are  controlled  either  by  the  student  or  by  the  instructional  media 
device,  but  the  nature  of  the  desired  recall  behavior  does  not 
require  recall  under  such  conditions. 


1-1.4. 5  Display  Considerations 

Display  considerations  for  this  type  of  objective  are  simpler 
than  for  the  (symbol/name)  objectives  described  in  Section  1-1.3. 
Motion  is  not  required  and  graphic  illustration  is  normally 
suf  f ic lent  . 


1-1.4. 6  Response,  Scoring,  and  Feedback  Considerations 

Response  considerations  for  this  type  of  objective  are  the 
same  as  those  for  Remember/Procedure  (paraphrase)  objectives  as 
described  in  Section  1-1.2. 6. 


1-1.5  REMEMBER/FACT  (LIMIT/TOLERANCE/CRITICAL  VALUE) 


I-1.5.1  Objective  Form 

The  distinguishing  feature  of  this  type  of  objective  is  that 
it  requires  the  student  to  learn  an  isolated  fact  normally  appear¬ 
ing  as  a  specific,  individual  value.  During  the  planning  of  the 
F-16  mission  there  are  several  of  these  individual  facts  which  must 
be  known  by  the  pilot  and  objectives  must  be  written  to  ensure  the 
students  are  aware  of  these  values.  Objectives  of  this  sort  are 
worded;  "State  the  (value)." 


1-1.5.  2  F-16  Example 

Examples  of  this  objective  type  from  the  F-16  objectives  would 
be  those  asking  the  student  to  state  the  maximum  crosswind  limits 
for  takeoff  in  the  F-i6,  the  maximum  roll  speed,  the  minimum  roll 
distance,  the  maximum  wing  loadings,  and  so  forth.  More-involved 
objectives  of  this  type  may  require  students  to  state  weapon 
envelope  parameters  or  aircraft  operation  envelopes. 


1-1.5. 3  Strategy  Components 

Objectives  of  this  variety  are  critically  important  and  can 
affect  safety  of  flight  and  life.  Instr  uc  t  ional  ly  ,  however,  these 
individual  objectives  are  often  not  substantial  enough  to  stand 
alone  as  the  major  focus  of  an  instructional  presentation.  For 
instructional  purposes  these  objectives  will  probably  be  grouped 
together  or  else  be  combined  with  objectives  of  other  types  so  that 
instruction  and  practice  on  these  lesser  objectives  will  take  place 
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in  the  context  of  instruction  and  practice  on  the  more  inclusive 
obj  ectives . 

Because  objectives  of  this  variety  will  be  piggybacked  in  this 
fashion,  the  strategy  requirements  for  them  will  be  simple, 
i  n  c  1  ud  i  ng  ; 

Generality — a  statement  of  the  individual  value  to  learned. 

For  single  values  this  is  a  simple  sentence.  For  more 
involved  sets  of  values,  this  may  be  a  table  or  chart. 

For  many  of  the  more  involved  sets,  the  objective  will 
be  stated  as  a  Remember/Procedure  (paraphrase) 
objective  and  instructed  as  such. 

Practice--exercises  in  the  recall  of  the  value(s)  will  be 

mixed  in  with  exercises  for  the  other  objectives  which 
are  being  instructed  concurrently. 

Practice  help--See  Section  1-1.2.  3. 

When  instruction  on  these  objectives  is  embedded  in  other 
instruction  appropriate  emphasis  must  also  be  treated  as  an 
element  of  the  strategy,  both  in  the  instruction  and  in  the 
testing  which  follows  instruction.  This  emphasis  will  constitute 
a  major  part  of  the  instructional  strategy  and  can  be  obtained 
either  by  visual  effects  or  verbal  instructions  to  attend. 


I-1.S.4  Control  and  Pace  Considerations 

Control  and  pace  considerations  are  not  relevant  to  this 
objective,  since  instruction  will  take  place  in  the  context  of 
another  objective. 

1-1.5. 5  Display  Considerations 

Display  considerations  require  only  the  ability  to  state 
simple  verbal  and  sometimes  tabular  or  simple  graphic  data. 


1-1.5. 6  Response,  Scoring,  and  Feedback  Considerations 

Scoring  considerations  are  governed  by  the  controlling 
strategy  within  which  instruction  on  the  value  objective  takes 
pi  ace  . 


1-1.6  REMEMBER/PACT  (ORGANIZED  LISTS) 


I-1.6.1  Objective  Form 

Objectives  of  this  type  are  worded:  "State  the  _ ,"  where 

the  object  refers  to  a  list  of  events,  items,  obj ects~whTch  are 


in  some  fashion  related  to  each  other  by  some  principle  other 
than  sequence.  Though  there  may  be  sequence  in  some  parts  of  the 
list,  the  sequence  itself  is  not  the  organizing  principle  of  the 
list.  A  second  form  in  which  this  objective  will  appear  could  be 
called  the  interference  form  of  the  objective.  It  is  used  after 
several  objectives  of  the  type  have  been  mastered  and  when  it  is 
desired  that  the  information  pertinent  to  all  of  them  be  prac¬ 
ticed  together.  This  form  of  the  objective  states:  "Given  the 
learned  list  of  items,  name  the  phenomenon  which  they  indicate." 
This  is  very  close  to  a  use/concept  behavior  because  the  student 
is  asked  to  name  an  object  or  event  by  its  relevent  characteris¬ 
tics.  It  is  not  exactly  use/concept  behavior,  however,  because 
the  student  is  asked  to  respond  from  memorized  information, 
rather  than  generalizing  a  response  across  a  class  of  unencount¬ 
ered  objects. 


1-1.6.  2  F-16  Examples 

Examples  from  the  F-16  objectives  which  are  of  this  type 
include  those  which  ask  students  to  list  the  indications  of  a 
certain  emergency,  the  varieties  of  search  for  enemy  aircraft,  or 
the  main  principles  of  energy  management. 


1-1.6.  3  Strategy  Components 

Strategy  components  for  this  type  of  objective  will  include: 

General  ity--a  statement  of  the  elements  of  the  list  and  the 
definition  of  each  as  appropriate. 

Organizing  Principle  (Generality  Help  I) — a  statement  either 
verbally  or  graphically  of  the  principle  which  relates 
the  elements  of  the  list  together.  This  is  provided  as 
a  mnemonic. 

Mnemonic  (Generality  Help  II)--a  verbal  or  graphic  memory 
aid  artifically  contrived  for  further  use  by  the 
student  as  a  memory  aid  during  recall. 

Discussion  (Generality  Help  HI)  —  additional  information 
relative  to  each  item  on  the  list  either  (a)  elabo¬ 
rating  the  definition  of  each,  or  ( b)  providing  further 
memory-aiding  information. 

Prac t  ice--verbal  or  timed  drill  and  practice  exercises  as 

described  in  previous  sections.  No  priority  is  placed 
on  the  timed  exercises,  and  they  should  be  used  only  if 
a  media  device  is  available  which  can  administer  such 
exercises  . 

Practice  help--Gee  Section  1-1.2.  3. 
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1-1.6. 4  Control  and  Pace  Considerations 

Control  and  pace  considerations  for  this  variety  of  objec¬ 
tive  are  identical  to  those  discussed  earlier  in  Section  I-1.1.4 
and  1-1.2. 4,  whichever  applies,  since  either  machine-  or  student- 
administered  instruction  and  practice  may  be  used. 


1-1.6. 5  Display  Considerations 

Display  considerations  for  Rem  ember /Fact  (organized  list) 
are  restricted  to  verbal  or  simple  visual  presentations. 


1-1.7  REMEMBER/FACT  (SEQUENCE)  OBJECTIVES 


I-1.7.1  Objective  Form 

Objectives  of  this  type  are  similar  to  Rem  ember /Rr  oced  ur  e 
(paraphrase)  objectives  in  that  the  context  being  learned 
consists  of  steps  in  a  sequence.  The  major  difference  between 
them  is  that  Remember /Proced  ure  (paraphrase)  objectives,  state  a 
sequence  to  be  remembered  made  up  o  f  steps  the  student  will 
perform.  For  Remember/Fact  (sequences)  objectives  the  sequence 
is  not  one  which  will  be  performed  by  the  studeht  but  one  which 
the  student  is  expected  only  to  observe  and  recall.  In  such 
objectives  the  interrelation  of  cause  and  effect  between  steps  is 
often  an  important  feature  of  the  sequence. 

The  wording  of  Remember/Fact  (sequence)  objectives  is: 

"State  the  sequence  of  _ ". 


1-1.7. 2  F-16  Examples 

Examples  of  this  type  of  objective  from  F-i6  would  be 
objectives  requiring  the  student  to  know  the  operational  sequence 
of  any  aircraft  system  or  sub  system,  such  as  the  radar. 


1-1.7. 3  Strategy  Components 

The  strategy  components  for  this  type  of  objective  will 
incl ude  : 

Generality — a  verbal  statement,  with  appropriate  graphic 
support,  of  the  sequence  to  be  learned. 

Supporting  Detail  (Generality  He  1  p) --add  i  t  ional  information 
on  each  step  of  the  sequence  emphasizing  the  cause  and 
effect  flow  from  one  step  of  the  sequence  to  the  next 
and  the  interrelations  between  elements  involved  in 
each  step  of  the  sequence. 
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Practice — appropriate  verbal  exercises  such  as  those 
described  in  Section  1-1.2.  3. 

Practice  Help--See  Section  1-1.2.  3. 

1-1. 7. 4  Control  and  Pace  Considerations 

See  Section  1-1.2.  4. 


1-1. 7. 5  Display  Considerations 

Motion  is  not  required  for  displays  for  this  variety  of 
instruction,  but  it  can  be  looked  upon  in  many  cases  as  a 
facilitating  factor  for  student  uptake  of  the  information  being 
presented.  If  motion  is  not  possible,  still  graphic  displays 
should  suffice. 


1-1. 7. 6  Response,  Scoring,  and  Feedback  Considerations 

See  Section  1-1.2.  6. 


1-1.8  USE/CONCEPT 


1-1. 8.1  Objective  Form 

Use  concept  objectives  are  those  objectives  which  have  the 
form:  "Given  a  previously  encountered  instance  or  noninstance  of 

a  class,  state  whether  it  is  a  member  of  the  class,"  or,  "Given  a 
series  of  items  previously  unencountered  as  exempliers  of  any 
given  class,  identify  each  by  the  name  of  its  class." 


1-1.  8.  2  F-16  Examples 

Examples  of  this  type  of  objective  for  F-16  will  not  be  the 
most  nunerous,  but,  in  aircrew  training  applications  they  tend  to 
be  the  more  important  objectives.  Specific  examples  will  deal 
with  the  interpretation  of  radar  screens,  and  the  classification 
of  tactical  situations  by  type  of  strategy  planning  purposes. 


1-1. 8. 3  Strategy  Components 

Strategy  components  for  this  type  of  objective  include: 

Generality — a  verbal  definition  of  the  class  of  things  the 
student  is  learning  to  classify. 
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Elaboration  (Generality  Help  I) — an  extended  description  of 
each  main  attribute  of  the  class  definition.  This 
ensures  that  non-comphrehens ion  is  not  due  to  inade¬ 
quate  prerequisite  terms  and  concepts. 

Attribute  Isolation  (Generality  Help  II) — a  walk-through  of 
the  classification  of  one  example  showing  the  applica¬ 
tion  of  the  definition  and  identification  of  each 
attribute  separately. 

Example--the  presentation  of  a  member  of  the  class,  some¬ 
times  matched  with  non-members  for  emphasis. 

Emphasis  (Example  Help) --an  answer  to  the  students'  possible 
question  "Why  is  this  an  example?"  For  each  example 
this  display  will  be  created  showing  the  relevant 
elements  of  each  example  that  cause  the  example  to  be 
an  example. 

Practice--a  drill  in  which  the  student  is  asked  to  identify 
examples  and  non-examples  of  the  class  on  his  own. 

Practice  Help--a  display  for  each  practice  item  showing  why 
each  practice  item  was  or  was  not  a  member  of  the 
class . 


1-1.8. 4  Control  and  Pace  Considerations 

Control  and  pace  for  this  type  of  instruction  may  reside 
with  the  student  or  with  the  instructional  device.  There  is 
research  evidence  (see  Walker,  1977,  Chapter  III)  to  show  that 
students  given  free  access  to  the  strategy  components  and  allowed 
to  select  from  them  in  any  order  complete  their  instruction  in 
less  time  and  with  better  mastery  of  the  material.  A  media 
device  which  allows  this  type  of  student  control  should  also  be 
able  to  administer  device-controlled  instruction.  In  the  absence 
of  a  free  control  training  device,  instructional  order  can  be 
built-in  to  instructional  presentations. 


1-1. 8. 5  Display  Considerations 

Display  considerations  for  this  type  of  objective  include  a 
requirement  for  graphic  representation,  occasionally  high  detail, 
and  a  heavy  verbal  component. 


1-1.8. 6  Response,  Scoring,  and  Feedback  Considerations 

Response  capability  need  only  consist  of  a  multiple  choice 
selection  mechanism  for  concept  learning.  Scoring  or  responses 
may  be  machine-  or  student-adm  inister ed . 
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1-1.9  USE/RULE 


I-1.9.1  Objective  Form 

Use/Rule  objectives  are  worded:  "Apply  the  rule,”  where  a 
rule  is  a  series  of  steps  in  a  procedure  executed  on  unencount¬ 
ered  and  changeable  input. 


1-1.9.  2  F-16  Example 

Examples  of  F-16  rules  are  computational  rules,  or  general 
principles  used  to  determine  specific  solutions,  such  as  in  a 
tactical  problem. 


1-1.9.  3  Strategy  Components 

Strategy  components  used  for  Use/Rule  instruction  are 
essentially  the  same  as  those  for  Use/Concept  instruction: 

Generality — a  statement  of  the  rule. 

Elaboration  (Generality  Help  I) — a  walk-through  of  the 
application  of  the  rule  to  one  problem. 

Example — presentation  of  further  worked  examples  using  the 
r  ul  e . 

E!nphasis--a  walk-through  for  each  example  of  the  individual 
steps  in  applying  the  rule. 

Pr actice--problems  to  which  the  student  may  apply  the  rule. 

Practice  Help — a  walk-through  for  each  practice  item  of  the 
correct  application  of  the  rule  step-by-step. 


1-1.9. 4  Control  and  Pace  Considerations 

The  considerations  described  in  Section  1-1.8.  4  for  Use/ 
Concept  instruction  apply  to  Use/Rule  instruction  also. 


1-1.9. 5  Display  Considerations 

Considerations  listed  in  Section  1-1.8. 5  for  Use/Concept 
instruction  apply  here  also  with  the  additional  requirement  for 
occasional  display  of  tabular  material. 
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